tl;dr; When upgarding from Ubuntu 12.04 to 14.04 LTS, the Apache also gets upgraded from 2.2 to 2.4. There are lots of backwards incompatible changes.
I just spent the last 3 hours fixing my server after impulsively upgrading to Ubuntu 14.04.1 LTS from Ubuntu 12.04.
Well, not exactly that impulsively. I had seen this pestering MOTD-style message upon login for weeks now, and I figured that since 14.04.1 is a point release nearly 6 months after the initial release, it should be relatively issue-free.
I had already done 3 upgrades from 12.04 LTS to 14.04 LTS… on desktops. Two had gone successfully, and one on my parents’ computer unfortunately messed up pretty badly in the middle so that some of the ubuntu-desktop stuff is just not working properly, but I digress.
I figured that since it was a Friday night in addition to the above–“What the heck, why not?”–and dove right in. The OS upgrade was pretty much issue free. I let out a sigh of relief.
And then my luck ran out. All of my websites didn’t load. Read: Nothing worked. I started getting Pingdom pages and Nagios alerts non-stop.
Thankfully, the Internet is a great resource and I was able to identify the problem pretty quickly. Applying the fix is what took most of the time.
Most useful of all the resources that I came across was Upgrading Apache 2.2 to 2.4. I’d recommend reading through it thoroughly if you’re going through the upgrade yourself, but here are some highlights:
- Don’t be alarmed if all you’re seeing for any of your websites is just the contents of
- All separate virtual host (
VirtualHost) files and configurations need to end in
*.confdue to the command in
- It still wasn’t pulling in my VirtualHost
- Got rid of
NameVirtualHost *:80as the first line in all of my VirtualHost config files. It wasn’t necessary. Good riddance.
Directorypermissions now need to be explicitly granted.
Require all granted(the old style was
Order Deny, Allow; Allow from all)
- I noticed that my websites were loading slowly, so needed to set
EnableSendfile Onin the main config file (
- For my Django sites, I needed to change the
Directorypermissions for the static files directory
- For my custom WordPress sites, I needed to set the
Directorypermissions for the
DocumentRootas well as explicitly set
AllowOverride allto allow picking up the
short_open_tagdisabled. I decided to leave it off and change my limited number of PHP applications that were using short open tags, because it’s better practice.
Other helpful resources:
If this is you:
- Android developer with an Intel-based computer
- Sick of dealing with a slow emulator
I never knew this, and thought that I was doomed to a slow emulator or had to debug and deploy over a constantly plugged-in device, which could at times be inconvenient.
Got an email today about a new Android training program starting up in SF for engineers. Best of all, it’s free!
I’m definitely going to check them out and try to sign up.
Update (2013-09-15): If you’re unable to attend the class, they may have a “course observer option where you will have access to all the materials for the class, but you just won’t participate in the live classes, discussion forums, or the group projects.”
I’m a co-founder of CodePath, a startup that runs iOS and Android training programs. We’ve trained hundreds of engineers at Yahoo who are moving from web development to mobile development.
We’re offering the same training program to engineers in the bay area for free. It’s a project-based course, meets twice a week for 6 weeks, and the next class begins on October 2nd (another one starts in January). It’s an intensive program, but we’ve seen great results with it.
We’re currently accepting engineers with a technical degree OR at least two years of object-oriented programming experience. If you’re interested, sign up here:
- Android class, starting October 2
- Android class, starting January 15
- iOS class, starting October 2
- iOS class, starting January 15
Please forward to any engineers or mailing lists that might be appropriate, additional details below.
Additional Program Details
This is an evening bootcamp meaning that all sessions are held after work hours and all of the bootcamp projects can be done after hours and on the weekends. Here’s a summary of what you need to know:
- Free of charge for engineers
- We will select up to 30 applicants for each cohort
- 6-week evening bootcamp starting Oct 2nd
- 2 on-site sessions each week in the evening (instruction and lab, 7-9 pm)
- Sessions held at Zynga SF office at 8th and Brannan
- A mini-app is built each week (5 total projects)
- Group project designed and developed over 6 weeks
Here are the requirements for the course:
- Physically present in San Francisco and available Mon+Wed from 7-9 pm during sessions
- Technical (CS, EECS) Degree and/or 2 years of professional software development experience
- 8-10 additional hours every week to dedicate to projects and collaborative learning
- Existing development experience with object-oriented languages
- Passionate about learning and collaborating on mobile projects
- Laptop with Mac OS X (iOS, Android) or Windows (Android)
Seriously though, I messed up in my previous blog post. Pinterest wasn’t hacked, and I’m sorry that I jumped to conclusions when I actually stumbled upon a spammer’s C&C. I still feel that the actions I took at the time were correct according to the knowledge that I had, but based on what I know now, I would have approached it different (contacted Pinterest first, now that I actually have their contact info, before writing a blog post).
Thankfully, Jon Jenkins from Pinterest did contact me:
I lead the engineering team at Pinterest. I saw your blog post you wrote about Pinterest security.
For the record the site you found is not one that is associated with Pinterest in any way.
It appears that you most likely found a command interface for a Pinterest spam net. We have systems that detect and mitigate the impact of these spam nets. It would be convenient if you could pass along the addresses and passwords you obtained so that we can validate our spam detection systems are picking up these accounts.
And my response:
Thanks for reaching out to me. I later realized based on the discussions and Hacker News comments that this wasn’t Pinterest (although, there was still a small possibility that it was an internal tool written by a small team within Pinterest, or by an individual employee for a Hack Day contest, etc). I’m sorry for jumping to conclusions and accusing Pinterest of security vulnerabilities, but I had thought that just as if my friend felt he was having a heart attack, I would call 911 first and then wait for diagnosis later, rather than let the window of urgency elapse. I’m planning on writing a follow-up blog post on that matter.
Anyway, regarding the collected addresses and passwords, I forwarded them to firstname.lastname@example.org, not sure if that was the correct email address or you picked it up. I just forwarded it to you again.
The majority of the spam accounts came from 3 domains, omitted, although there were two accounts that weren’t from those domains (omitted)
Also, not sure if it would be worth your time and effort, but maybe Pinterest could issue a subpoena to AWS for the information of the accounts/human users behind that spam net, or simply work with the Spam/Abuse teams at AWS to make sure these guys get shut down. Building systems to detect and prevent spam is a never ending cat and mouse chase, but if we could shut down some of the human elements, that might stir a greater effect?
The original discovery happened in the window of about 30 minutes to an hour before I posted this tweet:
The Elastic IP address of the C&C during that window:
Jon’s response to my response:
Thanks for passing along the email addresses. It’s always useful to get a known bad set of accounts to verify our classifiers are working as intended.
I was planning to write a blog post pointing out some of the problems in your original post. However, if you are going to correct things that might be good enough.
Regarding the specific claims you made in the original post: 1) We salt and encrypt all the passwords in our systems; 2) Our admin servers are protected in a number of ways in addition to those you suggest in your second point, 3) Regarding your third point we use more than just passwords to restrict access to our admin applications.
To everyone who made comments (constructive and otherwise) on the original post and on Hacker News, thanks. To the critics and the snide commenters, thanks for making me laugh at my own mistakes. I’m interested in learning about what others would have done differently if they were in my situation. Should I have spent more time collecting data about the spam C&C? What kinds of data?
What kinds of action can be taken against spammers like these? Is it worth having lawyers issue cease & decist letters and takedowns to the owners of those domains, knowing they can just as easily pop up in more places?
I certainly don’t have the time and resources (and motivation) to proactively fight against spammers, but I was hoping that Pinterest and AWS might. Also, this was a good reminder (similar to things I’ve heard in a security talk a few years ago by Billy Rios) that all the spammers and malicious hackers out there are just regular programmers like you and me. They aren’t necessarily super geniuses, they use pretty much the same languages and tools we do, and as all humans do, they make mistakes (like letting anyone with the IP address access the C&C).
It would also be interesting to see Amazon improve the way they are currently giving out Elastic IPs. AWS probably shouldn’t reassign recently released Elastic IP addresses and delay recycling for as long as possible, like how telecom companies don’t reassign discarded phone numbers until 3-6 months afterward. They could use an LRU list for assigning Elastic IPs out of regions–unless Elastic IPs really are that scarce a resource in high demand that even an LRU implementation would cause an Elastic IP to effectively get reassigned right away?
Update 2013-05-12: Pinterest wasn’t really hacked. See follow up to this blog post.
Well, almost hacked. This is rather embarassing (for Pinterest, and maybe AWS?), in that I was able to access what seemed to be their admin page. Furthermore, I discovered through this interface that it seems they do not store passwords encrypted or salted. If I’m not mistaken, I saw usernames, emails, and passwords in plaintext. Let me describe what happened.
I was migrating a Nagios monitoring server, which involved decommissioning the existing EC2 instance and releasing the Elastic IP. Because I didn’t delete my Route 53 DNS entry, and left the Nagios URL open in my browser, I soon noticed that some other content had taken the place in my browser window.
Basically, the apparent Pinterest admin server had requested an Elastic IP from the useast region, and had associated itself to the IP address that I had just released.
A few lessons can be learned from this:
- Pinterest should encrypt and salt their passwords, shame on them for not doing so.
- Pinterest should put their admin server so that it is only accessible from behind a firewall. Ever heard of EC2 security groups? I set up all my servers to expose only certain ports to certain other security groups. Only the external-facing webapp servers have port 80 open to the world.
- Pinterest should put a password on their admin site. Even a shared password internally on the HTTP server is better than having nothing at all.
A good hacking strategy would be to allocate associate disassociate release Elastic IPs, and to squat on/poll that IP address via several common ports. I got lucky and found one immediately through a browser.
Want proof that I actually did this? How about a screenshot:
My friend tells me that I should also mention that I was able to log in with the emails/passwords leaked. I saved around 37 of them, and tested successful logins for 2 of 2 that I tried. For discretionary purposes, those credentials will not be posted here. However, I will say that those accounts I saw in the admin interface seem to be fake hacker accounts, with the emails being mostly from the same domains.