In a Galaxy Far Far Away

As many of you already know, at the beginning of the year I joined Oak Ridge National Laboratory as a Research Associate. My job is to use my knowledge in HPC performance analysis to enable scientific codes (like these) to run better on Titan - the fastest open-science super computer of the western world and its successor Summit (POWER9+NVIDIA Volta). Needless to say that this is super-exciting work!

As a result, I now live in Knoxville/Tennessee and there is so much to talk about :). This is not a coherent post. More like a fact and opinion rapid-fire.

The flag of Tennessee, where it is on the map, and the ORNL Logo

Everything feels around 1.5x as expensive as in Germany. Around what I would expect if the exchange rate to Euro would have stayed where it initially was. Gas costs 53¢/l. Electricity costs 9.8¢/kWh. Let that sink in for a second - Electricity and gas are practically dirt-cheap. A full tank costs me $23. No wonder people drive these giant-ass pickup trucks.

Houses are generally cheaply made - starting at ~$50k. In my apartment the air conditioning system both cools and heats. This combination of cheap housing and super-inefficient heating resulted in my first two month's electricity bills being around $40 (~400kWh). And this has to be just heating and maybe the fridge (It's a biiiig fridge). Although electricity is cheaper, I managed to pay more than a single-person household would pay in Germany. Electricity is so cheap people don't even bother getting LED light bulbs.

Wow, just wow.

Meat, especially beef is a lot cheaper. You can get a decent steak for $15 per kg. In general beef tastes a lot lot better than what you can get in Germany. It's seriously amazing. Beans and rice are also cheap. Ben & Jerry's costs $4-5, which is a blessing :).

A gigantic T-bone steak

Dinner time <3

My phone bill is around $30 for 20 minutes, 30 messages and 550MB, and Comcast wants $70 for 25/5MBit. Comcast has the worst customer service I have ever seen. I spent many attempts and a total of 2-3 hours to get two things sorted out. It's actually ridiculous how unhelpful their customer support is.

The typical rent in Oak Ridge/Knoxville ranges between $500 and $1000 for a normal sized flat.

Spatially, everything here is very very spread out. Knoxville has a population of 185k on roughly Dresden's area, and Knox County has 430k on five times that area. Oak Ridge has 30k people on 3/4ths the area of Dresden. In order to go downtown I need to drive about 8km. To the next mall I need somewhere between 2 and 8km depending on which. The only thing where I can really walk to is the gas station down the road. Sidewalks are rare, almost non-existent. Only downtown is it possible to get anywhere without a car. That makes having a car essential. This also means you cannot run, bike or take a decent walk from your door step, but rather have to drive somewhere and then go from there. It's annoying having to drive, go for a run, and then drive home all sweaty.

So I had to buy a car. The best option turned out to be to just buy it from a friend who left the country at the time I arrived. It's a shiny 2015 Ford Focus Sedan 6-shift manual.

When I arrived in the USA, I used a rental car for while. During that time I came to like automatic transmission. Sadly this one is a manual. Well, at least no one will be able to steal it - although.

I think Focus sedans don't exist in Germany. Overall there are more big cars on the road here and most of them are sedans and pickup trucks. Hatchbacks are very rare. Almost no one drives small cars. Very rarely do you see a Smart, a Fiesta or a Golf.

I bought a dash cam to play around with and for insurance purposes. People here are bad drivers: No signaling, random lane switching, overtaking on the right. Something like the TÜV does not exist. There are cars driving around where you are surprised they are able to drive at all. This also means there is no emission checking. I saw couple of cars that emitted pitch-black fumes. What the hell. Getting a driver's license is very easy, and costs just $30. Many don't even have car insurance. Phoning while driving is allowed. Texting is not.

In Tennessee, you can film anywhere where people have no reasonable expectation of privacy. Which is (almost?) everywhere in public. Everywhere else you are allowed to record conversations of where you have at least the consent of one involved party, for example yourself.

When you go shopping, everything is put into bags for you by the cashier. Bags bags bags. One notable exception is Aldi. Here's a picture of the result of my first shopping tour in the US.

No one recycles. In order to recycle you need to drive to the next recycling center - 7km for me. It's just a simple place with big containers for each kind of waste. Before I began recycling, I created at least four times the amount of trash I generated in Germany, which is crazy.  Separating biodegradable waste is not a thing at all.

Many stores are open every day and for way longer than in Germany. For example at Walmart you can go shopping 24/7.

Tennessee sees extremely quick weather changes, I think because it's continental and the mountain ranges are vertical so that weather coming in from north and south passes through quickly. One day it's 0°C, the next day it's 10°C and a few days later it can be more than 20°C. And then suddenly a thunderstorm with 1cm hails lasting for just one hour shows up.  Ughh. I feel like I have seen more extreme weather in four months than my hole life.

Here's nine people blowing leafs around the apartment complex towards their truck. I would expect rakes, but no.

Another peculiarity is beer. The old opinion that American beer is bad does not hold anymore. Actually, I think it's overall better than in Germany. Mainly because the variety is so large. There are many small breweries creating all kinds of beers - IPAs, Ales, Stouts, Lagers, Pilseners. Their Pilseners are terrible in comparison to ours, though. One big downside is that I get headaches very quickly from their stuff. Not sure what exactly causes this. Perhaps it's the added sugar in some of them. The price for beer starts at around $1-1.5 for a 350ml bottle of industrial beer. A beer at a bar is usually between $4 and $6.

A bazillion beers on tap. Every time a barrel is emptied,
they add a different beer. Great!

They ship this to here!

Create your own six-pack for $10 at Kroger.

The German/Austrian beers at Bearden Beer Market.

Trivia is a big thing around here. Lots of bars do it on a weekly basis. Our moderator Will at Hops and Hollers does an awesome job at it. Trivia is awesome! You have to collectively find the right answer, and betting-rounds or be-as-close-as-you-can-to-the-correct-answer increase tension and fun. Since I'm here we managed to win twice, and perhaps we will make it to a second-tier tournament later this year. Our strong suits are science, music, movies, history, religion, alcohol, geography, everything German / European. We are reasonable good at sports and suck a lot at Celebrities, TV and commercials.

Imperial units. Some call them freedom units or retard units. Only a handful of countries still use them. I got used to them more quickly than I expected. You only convert when you need to be exact. Most of the time you don't convert at all. On the road 10 miles is far, a quarter mile is soon, 1000 feet is sooner and 100 feet is very soon. An ounce is 30 grams. A pound is half a kilogram. A quarter pound is 100g, 12 fluid ounces is a can, a freedom pint is 16floz which is around 500ml. Yes, an American pint is really smaller than an imperial pint. A gallon is four liters.

While I'm here I have the goal to try as much local things as possible. I tried Pop Tarts, Twinkies, steak, philly cheese steak sandwich, mac & cheese, burger, hot dog, pizza, salsa, tortillas, tacos, quesadillas, chicken wings, and all those food chains like Chipotle, Five Guys, Blaze Pizza, Wendy's etc. There is actually quite a few things that are super-tasty!

Twinkies do have an expiration date!

Among other things I also want to see a rocket launch, go to a republican rally and to the shooting range. That's going to be interesting!

Since the election of Donald Trump politics has been a big shitfest. While Germany wrangles over sharing passport data with the police and meta data retention, things like these happen here:

• Immigration-limiting executive order
• Bill that allows ISPs to sell your browsing history
• Companies might soon be able to force genetic tests on employees and share the results with third parties. 
•  Trump's budget proposal would cut science by $50B and increase military spending from $550B to $600B.
• JPL scientist had to unlock his work phone at the airport. The aftermath is a big discussion on how to travel with electronics to and from the USA. 
• Executive order to remove the privacy-rights of foreigners living in the US 
• Executive order to ignore climate change in favor of helping coal and oil

I hope that voters pick up on these things and react accordingly.

Oak Ridge National Laboratory, short the lab, is a very interesting place to work at. It is located on a 40km² patch of land in the middle of nowhere surrounded by mountain ranges. To get to it you need to drive at least 10km. My commute from Knoxville is about 30km. The nearest airport is 45km away.

Historically it is one of the sites created for the Manhattan Project. More specifically, the lab was tasked with creating large amounts of weapons-grade plutonium.

During this time the city of Oak Ridge has been established.

Today the lab still has the old graphite reactor opened for tourists. Besides that it has a number of leading-edge science facilities. Some of them are:

• The Spallation Neutron Source (SNS), a giant nuclear research reactor used for neutron imaging and radiation experiments by hundreds of scientists from all over the world
• The High Flux Isotope Reactor (HFIR), a different kind of research reactor
• The infamous Building 5300 which is right next to my workplace - Classified things are going on in there. Read the Wikipedia article for more info.
• The Oak Ridge Leadship Computing Facility (OLCF)
• And many more

The OLCF houses Titan, the #1 civil supercomputer of the western world. In the Top500 there are two Chinese machines that score higher, but that's it. Titan consists of 200 cabinets, 18,000 nodes, 18,000 NVIDIA Kepler cards and 300,000 AMD Opteron cores.

The successor to it is Summit, which will be built by IBM and will contain POWER9 processors, NVIDIA Volta cards and NVRAM. It will deliver north of 100 petaFLOPS. Before Summit starts arriving later this year, right now we have a test setup consisting of POWER8 + P100 cards and local NVMe storage to gain experience with. Shiny!

The lab has a staff of around 4000 people, plus 3000 guest researchers, and 500 interns annually from all around the world. It is a very diverse work place. I could not find numbers, but I would guess that at least a third are foreigners. I know people from Belgium, China, Egypt, England, France, Germany, India, Kyrgystan, South Africa and Spain. It feels like almost everyone has a PhD, and people are generally super-smart and relaxed. I love it.

One of the main tasks of the national labs in the USA is the maintenance of the nuclear arsenal. That includes for example simulating the decay of the weapons.

The nature of interactions at the lab is a bit different from what I am used to. Oftentimes people don't directly answer/admit something. There is a kind of double-speak depending on the subject. I can't really put a finger on it or want to give examples. It's just how it feels. I think this is a product of the type of work some people are doing, the things some people know/suspect, the fact that you should assume to be watched, and the fact that suing people is a national sport. The amount of ass-covering going on is substantial, not just at the lab, but in general. For example there is a banner in my building thanking people for millions of safe work hours. And packaging of food or hot beverages often tells you not to be an idiot.

The lab is used both by the Department of Energy (DOE) and Department of Defense (DOD). The DOD is the parent agency of the NSA.

I don't know how it exactly works, but I think the DOE is a rather civil branch whereas the DOD is the military. One of the tasks of the DOE is to maintain and control the nuclear stockpile together with the DOD, so that not only one department has control over it.

The median income per person in the USA is around $30k, the median household is $50k. Tennessee is a bit below that. With a master's degree I am at more than twice the per-person-median and in the top 20%. I feel the income distribution is a bit worse than in Germany, especially towards the lower end. I feel that lab employees are overall fairly rich. Rumor has it that better positions pay way way better than e.g. higher positions in German public service. It's run like a private company.

To conclude this post, here are a few photos for you to enjoy:

A panoramic view of a few houses and the Interstate 40

Lake Shore Park is nice for running.

On our way to Ober Gatlingburg to go skiing

There is a Disney store in West Town Mall

It consists mainly of Star Wars merchandise and princess costumes.

Pi Day!

Authentic German cuisine. Haven't been there, yet.

A muskrat minding its business

The lab geese are having babies! Update: The nest is gone. Investigation pending.

I wonder whether these two have names. If not I will name them.
Perhaps they should have badges like any other employee.

A bit of Saxony far from home. Thank you Aldi.

Freedom sushi: Deep-fried, mayo-based sauce, creem cheese.
A friend ordered tea, and the first thing offered was sweet tea, haha.

Someone at the Lab drives this bad boy

I hope you are having a great time at home. So far I'm having lots of fun here. Make sure to hit me up if you are nearby. I'll try to be at the Super Computing Conference later this year. Hopefully, I'll see you there. Bye!