Seriously though, I messed up in my previous blog post. Pinterest wasn’t hacked, and I’m sorry that I jumped to conclusions when I actually stumbled upon a spammer’s C&C. I still feel that the actions I took at the time were correct according to the knowledge that I had, but based on what I know now, I would have approached it different (contacted Pinterest first, now that I actually have their contact info, before writing a blog post).

Thankfully, Jon Jenkins from Pinterest did contact me:

Hi Jon,

I lead the engineering team at Pinterest. I saw your blog post you wrote about Pinterest security.

http://www.jontsai.com/2013/05/11/aws-ec2-security-vulnerability-and-pinterest-hacked/

For the record the site you found is not one that is associated with Pinterest in any way.

It appears that you most likely found a command interface for a Pinterest spam net. We have systems that detect and mitigate the impact of these spam nets. It would be convenient if you could pass along the addresses and passwords you obtained so that we can validate our spam detection systems are picking up these accounts.

Thanks,

Jon Jenkins

And my response:

Hi Jon,

Thanks for reaching out to me. I later realized based on the discussions and Hacker News comments that this wasn’t Pinterest (although, there was still a small possibility that it was an internal tool written by a small team within Pinterest, or by an individual employee for a Hack Day contest, etc). I’m sorry for jumping to conclusions and accusing Pinterest of security vulnerabilities, but I had thought that just as if my friend felt he was having a heart attack, I would call 911 first and then wait for diagnosis later, rather than let the window of urgency elapse. I’m planning on writing a follow-up blog post on that matter.

Anyway, regarding the collected addresses and passwords, I forwarded them to engineering@pinterest.com, not sure if that was the correct email address or you picked it up. I just forwarded it to you again.

The majority of the spam accounts came from 3 domains, omitted, although there were two accounts that weren’t from those domains (omitted)

Also, not sure if it would be worth your time and effort, but maybe Pinterest could issue a subpoena to AWS for the information of the accounts/human users behind that spam net, or simply work with the Spam/Abuse teams at AWS to make sure these guys get shut down. Building systems to detect and prevent spam is a never ending cat and mouse chase, but if we could shut down some of the human elements, that might stir a greater effect?

The original discovery happened in the window of about 30 minutes to an hour before I posted this tweet:

http://twitter.com/jontsai/status/333090050231373825

The Elastic IP address of the C&C during that window:

ec2-omitted.compute-omitted.amazonaws.com

Thanks,

jontsai

Jon’s response to my response:

Thanks for passing along the email addresses. It’s always useful to get a known bad set of accounts to verify our classifiers are working as intended.

I was planning to write a blog post pointing out some of the problems in your original post. However, if you are going to correct things that might be good enough.

Regarding the specific claims you made in the original post: 1) We salt and encrypt all the passwords in our systems; 2) Our admin servers are protected in a number of ways in addition to those you suggest in your second point, 3) Regarding your third point we use more than just passwords to restrict access to our admin applications.

To everyone who made comments (constructive and otherwise) on the original post and on Hacker News, thanks. To the critics and the snide commenters, thanks for making me laugh at my own mistakes. I’m interested in learning about what others would have done differently if they were in my situation. Should I have spent more time collecting data about the spam C&C? What kinds of data?

What kinds of action can be taken against spammers like these? Is it worth having lawyers issue cease & decist letters and takedowns to the owners of those domains, knowing they can just as easily pop up in more places?

I certainly don’t have the time and resources (and motivation) to proactively fight against spammers, but I was hoping that Pinterest and AWS might. Also, this was a good reminder (similar to things I’ve heard in a security talk a few years ago by Billy Rios) that all the spammers and malicious hackers out there are just regular programmers like you and me. They aren’t necessarily super geniuses, they use pretty much the same languages and tools we do, and as all humans do, they make mistakes (like letting anyone with the IP address access the C&C).

It would also be interesting to see Amazon improve the way they are currently giving out Elastic IPs. AWS probably shouldn’t reassign recently released Elastic IP addresses and delay recycling for as long as possible, like how telecom companies don’t reassign discarded phone numbers until 3-6 months afterward. They could use an LRU list for assigning Elastic IPs out of regions–unless Elastic IPs really are that scarce a resource in high demand that even an LRU implementation would cause an Elastic IP to effectively get reassigned right away?

Update 2013-05-12: Pinterest wasn’t really hacked. See follow up to this blog post.

–

Well, almost hacked. This is rather embarassing (for Pinterest, and maybe AWS?), in that I was able to access what seemed to be their admin page. Furthermore, I discovered through this interface that it seems they do not store passwords encrypted or salted. If I’m not mistaken, I saw usernames, emails, and passwords in plaintext. Let me describe what happened.

I was migrating a Nagios monitoring server, which involved decommissioning the existing EC2 instance and releasing the Elastic IP. Because I didn’t delete my Route 53 DNS entry, and left the Nagios URL open in my browser, I soon noticed that some other content had taken the place in my browser window.

Basically, the apparent Pinterest admin server had requested an Elastic IP from the useast region, and had associated itself to the IP address that I had just released.

A few lessons can be learned from this:

1. Pinterest should encrypt and salt their passwords, shame on them for not doing so.
2. Pinterest should put their admin server so that it is only accessible from behind a firewall. Ever heard of EC2 security groups? I set up all my servers to expose only certain ports to certain other security groups. Only the external-facing webapp servers have port 80 open to the world.
3. Pinterest should put a password on their admin site. Even a shared password internally on the HTTP server is better than having nothing at all.

4. A good hacking strategy would be to allocate

   associate

   disassociate

   release Elastic IPs, and to squat on/poll that IP address via several common ports. I got lucky and found one immediately through a browser.

Want proof that I actually did this? How about a screenshot:

–Edit–

My friend tells me that I should also mention that I was able to log in with the emails/passwords leaked. I saved around 37 of them, and tested successful logins for 2 of 2 that I tried. For discretionary purposes, those credentials will not be posted here. However, I will say that those accounts I saw in the admin interface seem to be fake hacker accounts, with the emails being mostly from the same domains.

I have contacted Pinterest through whatever means I could find, via a Twitter @Pinterest msg and #pinterest hash tag, and also emailing engineering@pinterest.com.

–Edit #2–

Discussion on Hacker News

I call myself a generalist, but feel that I’m more adept at backend development than web frontend, so I’m always amazed by the simplicity and cleverness of really cool HTML/CSS/JS techniques out there.

1. Responsive Sticky Footer by Timothy Long [demo] [code]

This was very informative. I have yet to use this solution (maybe when I need a dynamically-sized footer), but it seems to be very promising–not to mention that the solution is very clever.

1. CSSSitckyFooter [code]

I’ve been using this exact technique for several (dozens) websites, and it’s a tried-and-true method.

1. Another CSS Sticky Footer by Ryan Fait [code]

I haven’t used this one, but it’s neat to see the code in action and how few lines it requires.

On a separate note, I recently started using LESS, a dynamic stylesheet language which allows you to write less CSS by using variables and functions. I can’t emphasize enough that it’s a HUGE timesaver. It’s a must-have for any project.

On Sunday, February 3, 2013 (Super Bowl Sunday), Domino’s Pizza disabled ShopRunner free delivery on their site. This screenshot was taken from the Domino’s Order Page on Tuesday, February 5, 2013, when they added it back in.

Coincidence? Sneaky Domino’s. I didn’t think to take a screenshot at the time, though. Too bad!

Let’s see what sharing to Twitter yields…

Today, I was looking up biking directions to my new office. I wanted to take VTA Express from Fremont to Sunnyvale, and then bike the rest of the way.

So, I looked up biking directions on Google Maps, and, to have a better idea of the actual road conditions, looked at street view. That’s when I noticed something interesting–all the cars on the road look the same!

Am I seeing multiples?

The Google Streetview car was slightly ahead and to the right of the car in the picture driving down Java Drive, and therefore, that same car was pictured in every frame of map tiles along that stretch of road.

What appears to be an optical illusion is… an optical illusion created by stitching together several separate frames together.