Thursday, April 25, 2013

Miss Daegu 2013 Contestants Face Morphing

Recently, there is a debate on plastic surgery from the post on Reddit titled "Korea's plastic surgery mayhem is finally converging on the same face. Here are the miss Korea 2013 contestants.

To better see the similarity among them, I first took the images from one Japanese blog. After simple normalization and registration, I can get an aligned, animated GIF looping through the 20 contestants as shown below.

                                       

These images remind me of a course project in computational photography I did two years ago. I wonder how the "averaged" face of these 20 contestants look like. Thus, I morph all the face images into a mean face shape, then average over the picture values using the code I developed in the course project. Here is the result.


Using the shape of the face images, I can generate a short movie of face morphing from contestant number 1 to 20. We can now see how one face is smoothly transformed into another.





Another application is to synthesize faces with new shapes. For example, I can morph each face a bit toward the mean shape. This results in a stabilized version of animated GIF looping from contestant 1 to 20 as shown below. 





So far, the above visualization figures are qualitative, we may be able to get more insights by quantitative analysis. 


First, we construct the eigenspaces from these contestants faces. However, the standard Principle Component Analysis cannot be directly applied due to the slight pose variation and occlusion by the hair. Instead, we perform a robust version of PCA, e.g., [Robust PCALow-Rank Matrix Recovery], to factor the low rank part and the sparse errors. Specifically, I use the implementation provided in [Low-Rank Matrix Recovery].
[The Augmented Lagrange Multiplier Method for Exact Recovery of Corrupted Low-Rank Matrices, Z. Lin, M. Chen, L. Wu, and Y. Ma (UIUC Technical Report UILU-ENG-09-2215, November 2009)]
Here are three sample results.



Second, by performing singular value decomposition, we get the eigenfaces and the corresponding eigenvalues. Here we can see that the eigenvalues vanish after 7, suggesting that the rank of the image data is 6.

Here we visualize the largest 6 eigenfaces. These eigenfaces encodes the variation within the 20 contestants.




Third, by projecting faces onto the eigenspace, we can analyze how these faces are distributed in the eigenspace. We show here the coefficients of the 20 contestants. Note that most of the energies are concentrated on first two eigenvalues.
Thus, we plot the coefficients corresponding to the first two eigenvectors. We can now see how similar the appearance of these contestants are.



 Fourth, we can try to construct the affinity matrix which encodes the pairwise similarity. (blue indicates similar and red indicates dissimilar).


This affinity matrix facilitates many interesting applications. For example, by summing up over columns or rows, we get a measure of how a contestant is different from the rest. In the plot below, x-axis denotes the contestant index (from 1- 20). The y-axis indicates how distinct the contestant is. This measure can also be interpreted as the salient object among the 20 contestants using the surround center information divergence.
(If you are interested in saliency, please see our paper on ICPR or slide.)

From the plot above, we can see that contestant 1, 2 and 6 are more distinct (or more salient) than the rest of the contestants. On the other hand, contestant 7, 12, and 15 are more common (i.e., closed to the averaged face) than others. 

Salient contestants (1, 2, 6)



It is interesting that one actually used these most salient contestants (1, 2, 6) to support their opinions on how they are dissimilar [link]. (I am not sure whether they did the same eigen analysis or not.)

Common contestants (7, 12, 15)

Here are the pictures of all contestants.


Another application of using the affinity matrix is to come up with a displaying order with maximized or minimized similarity from one to another. You may use one of them to support your opinion on whether they are similar or not.

Consider a complete graph of 20 vertices with edges encode the pairwise similarity, the problem of finding a displaying order is the problem of finding a (weighted) Hamiltonian circuit in an undirected graph [see Hamiltonian path problem]. As the problem is known to be NP-hard, we could use a greedy heuristic to find approximated solutions. 

The heuristic algorithm works as follows. Start from a random contestant, choose the next contestant (that has not been selected before) with maximum or minimum similarity with the previous contestant. Repeat the process until we find the ordering.


Here is the result of ordering with maximized pairwise similarity. Note that the transition between contestants are more smooth.



Here is the result of minimized pairwise similarity. They look more dissimilar in this displaying order. 



Resources: 
  • If you would like to try it out on your own, you can download my code (in Matlab) here (12.7 MB)
  • You can also just download the cropped, aligned, and mean faces here (12.6 MB)
Have fun! :D



Further readings:



  • Average faces
    • Face of tomorrow project by Mike Mike. [link]
    • Face Research has a online demo for you to make an average face [link]
  • Face morphing [wiki]
  • Principle component analysis
  • Low-Rank Matrix Recovery and Completion via Convex Optimization [link]
  • Saliency
    • Salience (neuroscience) [wiki]
  • Hamiltonian path/circuit [wiki]

Updates (May 4, 2013)

1. The title of the original post was wrong. They are in fact contestants for Miss Daegu. I had corrected the title. Sorry for the confusion.

2. The poll results are out and is available on the official website


The top candidates are contestant number 5, 1, 15, 20, and 19.  We may be able to "guess" how the quantitative analysis of similarity is related to the result.

The first thing I observed is that one of the most salient (No. 1) and most common contestants (No. 15) won the 2nd and the 3rd places, respectively. This result seems to coincide with the psychological study on averaged faces. 

Thomas R. Alley and Michael R. Cunningham, Psychological Science, 1991

We then plot the coefficients corresponding to the first two eigenvectors for top 5 contestants (marked in red). It seems that the selection process somehow avoided choosing both candidates with similar appearances. 

Update (July 20, 2013)

One of the most salient contestant (number one), Yoo Ye-bin, crowned the Miss Korea 2013.

49 comments :

  1. Cool! Thanks for sharing interesting work.

    ReplyDelete
  2. AMAZING. I appreciate you sharing this.

    ReplyDelete
  3. Nice and fun project. Cheers.

    ReplyDelete
  4. Nice,
    However, I'd have been interested with a study of the difference with a group on "normal" (i.e. not models) Korean girls.

    ReplyDelete
  5. the candidates for this test should be proven that they underwent surgical procedure so we can see the "convergence" of their faces after surgery. also, sample pictures should have been makeup-less since they tend to use the same style of makeup and therefore be more similar especially in the eyes part.

    interesting, though.

    ReplyDelete
    Replies
    1. I agree. This post is very far from a serious study on the effect of plastic surgery. If more sample faces (under controlled lighting, poses, and w/wo makeup) are available, then we may be able to see more interesting results.

      Delete
  6. WOW this post is so coooooool!! Thought computer vision is not my major, I cannot understand it fully, I am considering whether it's possible to add a section to compare the average of the variance between the "normal people" and that between the "20 contestants." I believe we can see more interesting and inspiring results XD

    ReplyDelete
    Replies
    1. Yes, that will be an interesting comparison. Mike Mike had a project which computes average faces for people in different cities.

      http://www.businessinsider.com/faces-of-tomorrow-2011-2#altogether-now-23

      You can see the mean face of Korean near the top-right corner.

      Delete
    2. i think your talking about ANOVA (Analysis of statistical variance)

      Delete
    3. Although the comparison is interesting, it would only tell you about korean preference or what is considered beautiful in korean society, not anything about plastic surgery necessarily.

      Delete
  7. Makeups do make these pageants contestants look very similar rather plastic surgery. Here is the link of them before and after makeups:
    http://big5.xinhuanet.com/gate/big5/news.xinhuanet.com/photo/2013-04/28/m_124642965.htm

    ReplyDelete
  8. That explain my problem watching korean movies: I can't differentiate one girl from another. Men I have no problems; women look all the same to me.

    BTW, american porn have the same problem: all women have the same long hair, eyes makeup and overall face shape. It's the same face repeated thousand times.

    ReplyDelete
  9. It's not a real picture. They used Photoshop.

    http://pantip.com/topic/30421514

    ReplyDelete
  10. You want to rewrite the code in R for me? :)

    ReplyDelete
    Replies
    1. wow what a lazy ass! :)

      Delete
    2. http://stackoverflow.com/questions/5527145/convert-matlab-code-to-r

      Delete
  11. Replies
    1. science dude! fucking science.

      Delete
    2. I'm in Korea on business now haha... I keep falling in love

      Delete
  12. So, I have a question. Did one 7, 12, or 15 end up winning?

    ReplyDelete
    Replies
    1. my theory would be that, since they are all clearly trying to achieve an ideal, that the "most similar" of them should have the best chances, i.e. contestants 7, 12 and 15

      Delete
    2. The poll results are out and available on the official website
      http://misskorea.mpluskorea.com/missdaegu2013_poll

      Seems like both typical and atypical faces are popular.

      Delete
  13. 可以用它來作判定美醜的基準嗎,例如,

    1.與範本的近似權值,越接近 "理想值" 就越美...
    2.臉型眼眉鼻口相對位置與比例...整容業者會很想知道呢
    3.弄個 App 把照片 "自動美容"...應該會大受歡迎
    4.調整某些參數...拿來製作 美女臉蛋 的 3D 模型 ...

    ReplyDelete
    Replies
    1. There was a work on enhancing facial attractiveness using data-driven approach. This might be close to what you have in mind.

      http://www.youtube.com/watch?v=lVbrUuwK-8g

      Delete
  14. 可以用它來作判定美醜的基準嗎,例如,

    1.與範本的近似權值,越接近 "理想值" 就越美...
    2.臉型眼眉鼻口相對位置與比例...整容業者會很想知道呢
    3.弄個 App 把照片 "自動美容"...應該會大受歡迎
    4.調整某些參數...拿來製作 美女臉蛋 的 3D 模型 ...

    Agree!

    ReplyDelete
  15. Thus, we have mathematically proven that they arr rook same.

    ReplyDelete
    Replies
    1. Actually, I didn't prove anything in the post. We could not draw any conclusion from such limited samples.

      Delete
  16. Amazing. Nice to see practical applications of scientific analysis.

    ReplyDelete
  17. Very interesting and nice job!
    I am wondering whether you can derive how unusual
    their dis-similarity distribution is in a quantitive way?
    such as, the standard deviation of their distribution is much smaller than a random sample!

    ReplyDelete
    Replies
    1. Thanks! Yes, if we have some sample images of normal people in Korea, then, we might be able to see some interesting results. However, it may be hard to draw meaningful conclusion as we didn't control the shooting environment, e.g., lighting, pose, tone mapping.

      Delete
  18. Cool! You can even find out which surgery clinic is shared by different contestants.

    ReplyDelete
  19. 你好,请问一下对齐的脸是自己处理的吗?还是你拿到的数据就是脸部(尤其是双眼)已经对齐的了呢?如果是您自己处理的对齐,能否说下使用的方法吗?
    我从网上找了这些照片,用了一些方法检测到每张照片中双眼的位置,但是每个人双眼距离不同,所以双眼无法对齐,平均脸就看着不像一张脸了,非常想知道对齐方法,谢谢您!

    ReplyDelete
    Replies
    1. Hi Kevin,

      Nope, the raw data is not aligned.
      To align the images using eye positions.
      1. I manually labeled the eye positions for each faces.
      2. Compute mean locations of these eye positions.
      3. Compute similarity transformation from the eye positions in one image to the averaged position.
      4. Wrap each image according to the similarity transformation computed in (3)

      To compute the averaged face, I did something more. I first compute the mean shape. Then, wrap each image to the mean shape. Lastly, perform pixel-wise average.

      You may take a look of my code to see exactly how these steps are performed.

      Delete
  20. Yes, they have very talented hair and makeup people, but more importantly, which one had the best ballet technique? Which one played the concert harp? Which one wrote the best essay on the future of Korean feminism? Which one is an anthropology major and talented violinist? Which one is most likely to re-write your code?

    ReplyDelete
  21. Hi! This is really well done and smart working..I'm impressed much.
    Thanks you for sharing us.It's pleasure to follow you.
    Lots of loves from Turkey.

    ReplyDelete
  22. This comment has been removed by a blog administrator.

    ReplyDelete
  23. Good work thanks for sharing best Facial Plastic Surgery Post :)

    ReplyDelete
  24. Howdy! This is my 1st comment here so I just wanted to give a quick shout out and say I truly enjoy reading your posts. Thanks a lot!beauty surgery

    ReplyDelete
  25. This comment has been removed by the author.

    ReplyDelete
  26. Great work I enjoyed this post! I'm going to download and try the code now just for fun :) Thanks for sharing, really! Looking forward to more posts :)

    ReplyDelete
  27. Remarkable blog! I have no words to praise, it has really allured me.PerfectNose UK

    ReplyDelete