@Chuck Thanks so much for the incredibly detailed response! You're too kind. A few questions if you don't mind:

1) I read somewhere that some radar/sonar systems use the Doppler effect to measure distance/velocity of an object. Are those systems more uncommon than the one you described?

2) The goal is to make ball (e.g., golf ball, basketball, baseball, football, soccer) detection & tracking faster on smartphones by supplementing cameras with sensors. So wanted to explore if radar/sonar could help.

3) For the stereo system you describe, how far apart would the cameras need to be? What kind of range would a system with two cameras about 6-12 inches apart offer?

If you prefer to chat more via email instead of here, just lemme know. Thanks again @Chuck

@Crashalot - Some answers to your questions:

1) Doppler radar relies on the Doppler effect to do speed measurements. If you could imagine a radar wave as a series of peaks, then it might look something like |...|...|...|...|

With the Doppler effect, if the object is moving towards the radar transmitter, then one peak hits the objects and reflects, then the object moves closer to the transmitter in the inter-peak time. This means that the next peak hits the object sooner than it should have because the effective speed of the wave front was increased. This also means that the reflected wave returns to the transmitter sooner than expected.

So, where the transmit pulse looks like |...|...|...|...| , the return pulse might look like |.|.|.|.|

The opposite happens if moving away. This is like two cars on a street. If you (the wave front) are moving toward a car at 50 km/h while that car is also moving at you at 50 km/h, then it's as though the other car is stationary and you're moving 100 km/h.

The return signal is a higher (or lower) frequency if the object is moving toward (or away from) you.

This is how police radar guns work. They're also generally continuous-wave radar, which means they can't do any range detection, only speed.

Gotta stop here as I've got a big meeting to go to in real life, but I'll be back in ~1hr.

Back.

So, a pulsed radar system means you can emit a "ping" and then time how long it takes for the signal to return. This is an okay proposition if the speed at which the ping travels is relatively slow (sonar), but finding an accurate enough timer for a wave moving at the speed of light (radar) gets expensive.

Continuous-wave radar systems get around the cost of the timer by not using one. They emit a continuous signal (hence continuous wave) and then only look at the return frequency compared to the broadcast frequency. Now all that's required is a method to compare the difference between the frequencies - this is a common circuit used for FM radio demodulation.

Stationary objects don't affect the frequency of the signal (no Doppler effect), so they're ignored. A radar wave (like any wave) is periodic in nature, so it's not possible to do any ranging with a continuous wave because it's not possible to correlate a return peak with a particular transmit peak.

So, back to your questions:

2) I'm not aware of any ball detection on smartphones at all, unless you mean some camera-based tracker (but even then I'm not specifically aware of any particular app for this). One thing you may be able to utilize is the fact that sporting equipment is standardized. If you knew what you were looking at, you could use sport equipment standards to estimate the distance by comparing apparent size to the known size of the object.

You might get into trouble with baseball-softball, or something like an orange golf ball and a basketball, or similar situations, but again, if you knew in advance what it was, it should be a relatively straightforward thing to calculate.

3) I can't give specific guidance on stereo camera setups because I haven't actually implemented one before. I've got a lot of experience in actuation and control and did a master's thesis on radar (hence my ability to help out there), but I've never done anything more than basic blob detection and color thresholding with computer vision.

I will say that a stereo camera system relies on the parallax effect, where an object appears to be in a different location depending on the view point of the camera. Put your finger very close to your face, then look at it with one eye then the other. Your finger appears to move - this is parallax. At a "very far" distance away, there is negligible parallax. This is why the foreground moves but distant features (mountains, etc.) appear to be stationary.

What qualifies as "very far" depends on how far apart the cameras are located. You can increase the effective distance for resolving depth/distance by increasing the the distance between cameras, at the cost of being unable to detect objects close to the camera setup.

If you could imagine the field of view of one camera as a 'V', then two cameras close would look like VV. An object must exist where the Vs overlap in order to be given a distance estimate with stereo vision.

So, close Vs, like W, have a lot of overlap, but they're close, so the limit of their ability to resolve depth is pretty short.

Wide setups, like V V, can resolve depth for a long range, but there's also a long range before the Vs overlap, so there's a "dead zone" between the point of overlap and the camera setup where an object cannot be seen by both cameras and thus cannot be ranged.

Anyways, like I said, I'm not familiar with any smartphone app that does what you're talking about, so I'm still not positive I understand the point of doing it, but I don't know that purchasing a camera setup and a computer to do computer vision is necessarily the best answer to improving a smartphone app. Neither is radar/sonar, for that matter.

I think my approach would be to do background subtraction and a sports look-up table to compare image size to equipment size to estimate range. The x/y pixel coordinate (combined with the camera properties) would give the vector to the object, the size gives distance, and thus you can do 3d tracking of known objects with one camera.

You might be able to guess the object type by color histogram - white with some red would be a baseball, orange with some black is a basketball, brown and white is a football, anything else is a golf ball, etc. Leave the user a way to override this guess, tie the user/guess feedback to GPS location to help identify basketball courts, golf courses, etc., to improve guesses in the future.

Of course some things will get incorrectly guessed - colored basketballs, softballs, etc., but once you tag objects to locations (plus some region) then you probably get "good" at guessing pretty quick.

I'm fine with continuing the discussion here in chat. Moving to private email would feel more like consulting than free advice.

Wow, thanks again for the comprehensive answer! Yes, it increasingly seems like radar will not help. I was just exploring radars since self-driving cars use radar to help with "seeing" the road and potential obstacles so wondered if similar sensors could accelerate CV for mobile cameras.

Actually, I was exploring all types of sensors (e.g., thermal, radar, sonar) out of curiosity to see if any, combined with cameras, could short-circuit the time and processing for object detection & tracking. Unfortunately, it appears not for a handheld camera. @Chuck

At this point, the questions are more to round out my understanding of radar, if you don't mind:

1) Based on your very clear (btw you should teach, if you don't already) overview of radar, a pulsed radar system could not detect a ball behind a person because the pulse would hit the person first and return, and never detect the ball. Same with a continuous wave system (which can only detect velocity).

* Is that right?

2) Since timers for radar are expensive, does this mean there are no consumer radar devices that can do distance and speed? This device must use sonar to track distance? trackmangolf.com/products/trackman-4

Or that they must employ a continuous-wave system and can only detect velocity?

3) If stationary objects are ignored by Doppler systems, does this mean a Doppler system could track the velocity of a ball rolling around the ground? (you had said a pulsed system would have issues tracking a rolling ball. also we're assuming the ball is the only moving object in the scene.)

4) What is the "field of view" for a Doppler system?

5) Which schools & labs are known for the most innovative work on radar?

Thanks again @Chuck

@Crashalot - some answers:

1) If teaching paid what engineering paid I might. As it stands, I get to do R&D engineering work, which means I basically get paid (well) to do what I would probably do for free. It's a blast. I think teaching would be fun, but if I get paid double what my wife does (she's a teacher), that's hard to give up. But I've always said if I weren't an engineer I'd be a teacher.

Radar could theoretically detect objects behind/inside you, just like sonar (medical ultrasound) does, or X-rays (which are just a form of radio wave). The problem is at every "interface," which is the threshold where two dissimilar materials meet, a portion of the wave is allowed to pass and a portion is reflected. This "tax" applies in both directions.

So, if 80% was transmitted through you on contact, then 80% passes when it hits your skin and enters you, 80% when it exits you, then there's the reflection behind you, then 80% when it re-enters you, then 80% again when it re-exits you. So now, with nothing else taken into account, you've dropped the return strength by 60% just by being there. This is if the signal transmits that well through you, and doesn't count scattering on your internals, or transmission through clothing, etc.

Also, the ball won't return 100% of the signal, so you're only allowing 40% of whatever the return signal of the ball would have been.

Noteworthy too is the fact that each of those barriers does send a reflection, which is redirected back at the transmitter. The "best" or most significant (power-wise) response is typically the first one, all the rest of the reflections (bones, skin, organs, the ball, clothing, etc.) get very muddled very quickly.

Take a look at ground penetrating radar images. They're very difficult to analyze, even for trained forensic professionals. Certainly not a quick snapshot like a medical ultrasound might lead you to believe.

2) I haven't looked for consumer radar devices. The timers alone shouldn't be prohibitively expensive, but I would imagine they might be restricted as an international traffic in arms regulations (ITAR) object.

They might exist, they might not. No idea; I have never looked for them.

3) I would say probably not, especially for golf balls. Surface effects of radar systems get really weird. It's easy to think of radar waves as waves, and they act that way for the most part, but especially very close to a transmitter or near a surface they don't always act the way you would think.

Also, grass blows around, so there may be a fair amount of noise. It would be hard to distinguish the ball.

4) The field of view depends on the operating frequency. The operating frequency is generally selected based on the characteristics of the radio wave at that frequency. The atmosphere absorbs radio waves in particular bands, so those are generally avoided. Buildings absorb in some bands, vegetation absorbs in some bands, etc.

Also, probably more importantly, you can't just go operating radar in any band. You have to comply (in the US at least) with FCC regulations. Certain (most) spectral bands are restricted for particular use cases - TV bands, listening radio, police/emergency/military radios, medical bands, communications, etc. You run a real risk of knocking/jamming a bunch of equipment offline if you operate willy-nilly.

Some bands are called "ISM" (industrial, scientific, medical) bands. These bands are generally specified for anyone to use, provided you stay within the transmission power limits specified by the FCC.

These FCC limits apply to the US, but they've generally been adopted world-wide, but I would certainly check availability and licensing costs before going any further with the design.

So anyways, once you pick a frequency based on what you want and what's available, you decide how narrow you need the beam. Make a larger aperture (antenna) for a narrower beam, where large is measured in the number of wavelengths. Low frequency = longer wavelength = larger antenna to achieve the same beam width.

Radio telescopes for deep space observation operate at relatively low frequencies, and they want incredibly narrow beams to be able to resolve down to a planet at a distance of lightyears, so they setup whole fields of radio dishes to approximate a very large aperture. This is called a discrete aperture (as opposed to a continuous aperture, like one physical antenna).

Anyways, big antenna (aperture), narrow beam. This is generally pushed to the limit of practicality.

5) I have no idea. I was interested in building an imaging metal detector, and my thesis advisor let me pursue it. Quickly found out such a device is called radar, and my advisor still let me pursue it, so I did. But I never set out to design a radar system, I just did it kinda by accident. Huge learning process, and it was a great project, but not really related to most of the rest of my work. Just an odd thing I know a lot about now.

@Chuck thanks again! you're better than wikipedia! :)

you definitely have a talent for teaching

and yes, based on your explanations so far, it appears increasingly unlikely that adding radar sensors to a camera system accelerates CV (which is the ultimate goal)

so there are no commercial implications from learning about radar, but it's been incredibly fascinating. thanks so much for your time and effort! it's unfortunate teaching doesn't pay better because we need people like you educating people.