Discussion between A_A and TanMath

TanMath

6:19 AM

@A_A the RGB temporal signal for black:

TanMath

6:51 AM

@A_A RGB temporal signal for face image:

TanMath

7:05 AM

@A_A RGB face oscillating gif (sine wave):

TanMath

7:20 AM

@A_A face oscillating gif (square wave):

It seems that the red signal is attenuated.. but apart from that it seems this part is working... I think maybe my friend made the gifs like that...

A_A

3:35 PM

@TanMath This looks like a great start. I would suggest that you don't normalise. The transients in your graphs are coming from the camera auto exp algorithms. For the periodic waveforms, even if you don't apply ICA, it will simply work. You will get a peak at the expected frequency.

@TanMath The camera is dropping 1 frame every (approximately) 10 frames. Can you spot it? This might be problematic but not critical

@TanMath On the square wave the frame dropping is a bit more intense. Given what you have posted so far, this is a bit strange, but again, not critical.

@TanMath OK, the next step is to:

@TanMath 1) Remove the normalisation

@TanMath 2) Remove the bandpass filter

@TanMath 3) Use ICA and FFT steps

@TanMath 4) Check the output at the FFT stage

@TanMath Do these steps with the periodic waveforms

@TanMath This should work as expected

@TanMath The FFT should return a peak at the correct frequency.

@TanMath (Sorry, the camera might be dropping more than 1 frame, I didn't measure it accurately. What is the frequency of the periodic waveforms?)

@TanMath As a general comment: Given these specifc graphs, ICA has minimal effect because the signals are simply scaled differently.

@TanMath In other words, it might be better to actually extract 1 gray-scale channel by combining all three components to improve SNR but we will check again when you start feeding the system real videos as opposed to the phantoms you created.

@TanMath In any case, I don't know if you are doing this as a uni project or whatever but I would suggest that you add this part of the work as validation. It is very important to show how are you confident that the system works and what its limits are.

TanMath

7:34 PM

@A_A As a matter of fact, after you started recommending me to do this, I have been writing the procedures under validation. Yes, it is for a university final project...

@A_A Most papers normalize though... I think the reason it normalizes is because the ICA/PCA methods require normalization... So am not sure if dropping normalization all together is fine?

44

A: Why do we need to normalize data before principal component analysis (PCA)?

Normalization is important in PCA since it is a variance maximizing exercise. It projects your original data onto directions which maximize the variance. The first plot below shows the amount of total variance explained in the different principal components wher we have not normalized the data. A...

@A_A do you mean at the end it is not showing like an actual square wave?

A_A

8:26 PM

@TanMath For now, during testing, I would suggest no normalisation

@TanMath Yes, I understand, about ICA. The ICA function probably does its own normalisation already though.

@TanMath What I am trying to avoid with normalisation is "Jumps" in the signal. When you normalise, you normalise to some maximum. That maximum is obtained over a window. The window is finite. Therefore, from window to window you get these discontinuities as large "windows" of the signal get rescaled

@TanMath To avoid any confusion, the way you translate between FFT bin and physical frequency is $f = Fs \frac{k}{N_{FFT}}$ where $k$ is the FFT bin, $Fs$ is the sampling frequency and $N_{FFT}$ is the length of the FFT.

@TanMath Yes for the square wave. It is very pronounced there. In the sunusoid it is more subtle. Can you spot how does this jitter in the $Fs$ is affecting your signal? What is the effect it has on the signal without compensating for it?

@TanMath also, what effect do you expect it to have on your BPM estimation?

@TanMath You mentioned the Raspberry Pi earlier. Does it use a different camera? Is it a web cam? Is it USB?

TanMath

9:44 PM

@A_A is this with the sliding window or not? I will do it without the sliding window i guess...

@A_A Why include the ICA? isn't the ICA meaningless because what are the sources that it is separating?

the sliding window has 900 frames, but i recorded for 1100 frames...

TanMath

10:02 PM

@A_A give me two min i am almost done with the plots too...

A_A

@TanMath If you have a sliding window and normalisation then you will definitely get discontinuities

TanMath

@A_A Ok so first the sine waves:

And the square:

The FFT of the square wave is using the first ICA component as the other components weren't very oscillating...

And the FFT gave rubbish results...

@A_A OK these plots I did not use the sliding window and plotted/did FFT for the entire data...

A_A

10:18 PM

@TanMath This is normalised (?)

@TanMath If you don't normalise and don't use ICA the FFT should work fine

@TanMath Which module do you use for ICA?

TanMath

@A_A nope

@A_A oh ok... but yeah these plots are with ICA

@A_A sklearn.decomposition.FastICA

@A_A By any chance do you know why there is an oscillatory component in the FFT?

A_A

@TanMath I don't understand the question. That is what we expect to see... (?)

@TanMath You sample at 15 FPS, you enter a periodically varying waveform at some frequency f and (hopefully) you get to see it in the output of the fft

TanMath

@A_A 30 fps

or is this something else?

A_A

@TanMath Fs =30 FPS, f< Fs/2

TanMath

@A_A ok yeah... f < Fs/2 due to Nyquist sampling theorem...

A_A

10:32 PM

@TanMath Your FFT indicates something like 2Hz at it's strongest component . What is the frequency of the sunusoid?

TanMath

@A_A 2 Hz

A_A

@TanMath Then that part works.

TanMath

yeah...

A_A

@TanMath Now, bring ICA in

TanMath

@A_A sorry i didn't make it clear, but this was the ICA+FFT

A_A

10:35 PM

@TanMath OK

TanMath

7 hours ago, by A_A

@TanMath 3) Use ICA and FFT steps

A_A

@TanMath Yes, yes

TanMath

@A_A do you want the FFT by itself as well?

A_A

@TanMath Now, without using the filter, get a real measurement, with a face occupying as much of the camera FOV as possible

@TanMath BUT!!!

@TanMath The point here is that SNR is crucial. So, you need to find a way to light up the scene without saturating the camera

@TanMath Frankly, I think that it is the standardisation step that may be giving this method a boost in detection and not ICA

@TanMath That is, removing the mean and rescaling the amplitude

TanMath

@A_A What do you mean? Because the whole point of ICA was to separate a PPG-like signal from the RGB video...

A_A

10:42 PM

@TanMath You have now witnessed that your system responds as expected to a periodic waveforms.

@TanMath if you try to measure a real signal and you don't see measurable output, I think that it will be safe to assume that your SNR is very bad and the signal you are trying to measure is very weak

TanMath

@A_A but what would that be due to? the camera?

A_A

@TanMath The lighting will have a play into this too.

@TanMath The colour depth is 8 bit. That's 256 colours. I don't know what part of these 256 values will the PPG signal end up taking

@TanMath If the skin tone is 80 (for example) and the PPG causes a +2 on top of that, that's not going to be good

@TanMath This would mean that to get an accurate value at the very low frequencies you are trying to measure you would have to wait longer and accumulate more data

@TanMath To cut a long story short, your main problem is SNR. You need to check how much of the signal that is generated by the camera is due to PPG

TanMath

@A_A Ok so what I need to do is run with face detection, spatial averaging, ICA, FFT? no normalization, no bandpass...

@A_A but like we discussed earlier, the bandpass could also be a problem?

A_A

@TanMath Yes

TanMath

@A_A wait to both of the questions?

A_A

10:55 PM

@TanMath There is some delay in the messages. Yes, don't bring the bandpass in just yet

TanMath

@A_A ok... so you still think the ringing is a problem then... correct?

A_A

@TanMath Yes.

TanMath

@A_A i guess I never thought SNR would be a problem since the papers use not so great cameras in regular settings and got it to work, so I am wondering how they were able to do so...

A_A

@TanMath My guess would be that along with the visible, some of the IR is used too

@TanMath With what you have so far and what the papers mention, if there is a strong enough signal (associated with the PPG) then you should observe it

TanMath

@A_A should I do without the sliding window too?

A_A

11:01 PM

@TanMath If you don't observe it, based on the investigation we did here we would be more inclined to say that the signal is too weak to be observed

TanMath

@A_A should I also assume a sampling rate of 30 fps or should I measure it?

A_A

@TanMath Keep the sliding window. I don't know how it is implemented but I don't think it might be messing up the data

TanMath

@A_A I just do the calculations for the last 900 frames... so before, I used to do the normalization, ICA, and FFT for only the last 900 frames... but for the experiments you asked me to do recently, I did not do sliding window

@A_A and what about the sampling rate? what to do for that?

A_A

@TanMath What do you mean? In what sense?

TanMath

@A_A so like before I had taken the time measured at the 1st frame and the last frame of the sliding window to calculate the sampling rate... i am asking whether to keep that or to assume Fs = 30 fps

A_A

11:09 PM

@TanMath Let's assume 30 FPS for the moment. It doesn't really matter. What you expect to see is a strong periodic component that correlates with the heart bpm

@TanMath I have to go. Good progress. Check to see ways by which you could record the PPG related signal. If you observe a strong enough signal, the rest of the processing will pick it up

TanMath

@A_A wow... It seems like it might be working with these changes..

so the signal is there...

@A_A ok good night (i assume?)! i will post the results soon... thanks for the help!

@A_A i guess we might be narrowing down the problem to the bandpass filter...talk to you soon!

Transcript for

Discussion between A_A and TanMath