Prezly tech stack

A short description of technical components/stack that power Prezly and thousands of clients newsrooms. Components below are ordered in a way that made sense to me :-)

SSL/Termination: Nginx

Every application request will be greeted by Nginx. The job of nginx in this case is two fold:

  1. redirect non secure traffic to ssl (301 redirects)
  2. terminating ssl

For *.prezly.com we use a number of dedicated/wildcard SSL certificates. Client newsrooms using custom domains are served making us of letsencrypt certificates.

Nginx passes off requests to varnish or haproxy based on some hard coded rules. E.g. requests on rock.prezly.com are passed on to the load balancer directly because there is no use in caching them. 90% of requests is passed on to Varnish though.

Caching: Varnish

Frigging awesome caching layer. Prezly serves newsrooms for a number of large enterprise brands and so we encounter traffic peaks from time to time. Did anyone lose it’s aircraft ? More information on the new Porsche panamera ? Looking for awesome video’s on a Jetman next to an Airbus 380?

Luckily those peaks are highly cachable. We pretty much consider the bottleneck of Varnish to be the uplink from infrastructure set-up to the world. During large peaks I tend to log in to see how our load balancers are holding up. Most of the time the log shipping takes more resources then handling the traffic itself. Again, varnish is friggin awesome!

Did I mention it makes us lazy application engineers too ?

Load Balancer: Haproxy

Distributing the traffic to the different application servers is done by Haproxy. The number of active webservers is dynamic at all times but haproxy.cfg is modified automatically and hot reloaded by chef scripts upon server initialisation.

Requests are split up in a few categories/applications (api, backend, frontend, website) and those applications have different health checks. Upon health check failure (server failure, load problem, bad code,…) requests are passed on to the next available server. Kawabunga!

Webservers: Apache/PHP

To keep this post readable I won’t go into detail on application framework internals. So I’ll stick to stuff that fits in the context of this post.

  • Apache 2.4.7
  • PHP 7.0.7

The pool of active webservers consists of 4 machines. To minimise the monthly AWS invoice during weekends/downtime everything is being scaled down to a single webserver. Sessions are being shared using a memcached instance.

Recently upgrading PHP 5.x to PHP 7.x had a HUGE impact on performance. Response times for pages with some basic rendering logic/data fetching went from 74ms average to less than 50ms. Memory usage is a lot lower and the application performance is more predicable and stable. Where large application peaks used to result in a spin up of up to 4 webservers after upgrading I haven’t seen a load based spin up of over 2 web servers.

Workers: SQS/supervisord/PHP

We use a number of SQS queues to process background jobs. Those jobs are split up into different queues (p1 -> p4) which are handled by long-polling PHP scripts. PHP is daemonised by using supervisord where we spin up around 10 daemons per worker instance.

The good thing about this approach is that SQS supports long poll threading which has a great impact on the performance of the PHP daemons. Before using SQS we had a self-made queuing engine that used some kind of wait() function which caused the cpu load to go crazy during high concurrency.

Worker instances are spawned when the tresholds of the different queues are reached right up to the point where we start noticing slowdowns on database performance. Those rules are defined by using Opsworks auto-scaling rules.

Static Assets: Cloudfront

All images, attachments and videos uploaded by our customers are stored on AWS S3. We make use of cloudfront to surface those assets globally.

Database: PostgreSQL

In the past we have used MySQL, Mongo, CouchDB, MariaDB and probably a few others. Some of them were in production concurrently where we stored parts of our data in different storage engines. Sounds logical to me.

Today we use PostgreSQL only. Gotta admit, we’ve had our pain with background jobs, vacuum/analyse operations and IO tuning but we can say that database operations are under control now.

By making use of Amazon RDS we outsource the management and maintenance of our datastore.

Logging: rsyslog/elastic

For central logging we use rsyslog to ship logs to elastic search. Logs are being consumed using Kibana. I plan on writing a post the elaborates on the set-up of this central logging set-up.

Monitoring: 3 tools

We use new relic. I think it’s a little pricey for what we get out of it, but due to a lack of a better alternative we’re sticking around.

Because the new relic application pings/monitoring was returning too many false positives we traded that in for uptime.com which we are very happy with!

  • New Relic:
    • Application monitoring gives us a good insight in end-user performance and application performance
    • Server monitoring checks disk, cpu and memory usage and reports on that in slack
  • Uptime.com:
    • Global uptime
    • Domain renewals
    • SSL certificate checks
  • Cloudwatch:
    • Server monitoring including AWS performance indicators which is then used to auto scale the web servers.

All three services feed information to opsgenie and slack. Opsgenie then takes care of texting me out of bed when the shit really hits the fan (which never happened the last 2 years).

Potty training for tech entrepreneurs

In a lot of ways starting out as an entrepreneur is like being born. Your first months involve a lot of throwing up, doing things for the very first time and shitting yourself. It takes a while to get the basics right, eventually realising that there are better ways of getting from one place to another than flat on your belly.

The last years of my life have been primarily about those two topics: father-ship and entrepreneurship. Since then I’ve learned a ton, and wanted to take some time to share what i’ve learned and hopefully share an interesting story along the way.

Stop developing

We’re techies. The first solution to a problem is always that “i-can-build-something-for-that” attitude. Stop that!

As a technical co-founder we all carry the habit to start developing the solution. Actually, that next feature or that 16th integration will not get you the customers you are looking for!

If you can’t resist the urge to open that editor and start hacking away: focus on what you already have. Iterate on your existing features, make them faster, smarter and easier to use. Just stop producing more crap!

Learn how to sell

Reality check: 9 out of 10 startups fail. Most of them share the same pattern: Good people, high motivation and superb energy. But hardly enough focus on profitability and sales.

Successful startups are obsessed by sales in their earliest stages. Learn how to sell and how to deal with rejection. Try to add science into your sales process by learning about LTV, CPA, conversion rates and churn rates.

If you consider yourself a geek, confident around terminals and stack traces, throw out that new relic T-shirt and learn how to sell your idea. Start practicing on your pitch and get out the building!

Your brilliant idea is worth 20$

Screen Shot 2014-01-30 at 22.39.53

Still convinced that “finding the right idea” is the key to success ? It’s not. Execution is everything. There are so many ideas for businesses out there that can be successful with proper execution.

It’s the rework book that learned me about a simple table that displays that concept.

To make a business, you need to multiply the idea with the quality of the execution:

  • The most brilliant idea, with no execution, is worth $20.
  • The most brilliant idea takes great execution to be worth $20,000,000

Well, you get the picture

Choose your co-founders

Growing a business is hard. In a lot of ways it’s comparable to the first months of being a parent: waking up at night, coping with your inner-feelings about new responsibilities. Being tired most of the time will put some stress on your relation.

Co-founding a company much like a marriage (without the sex). For a considerable amount of time, you’ll likely be spending more time with your business partner than with your life partner.

Finding the right co-founders is among the most difficult, yet most important thing new entrepreneurs need to do. Find someone you can trust in the trenches, and share your drive toward constant improvement and success.

There is never a right time

Waiting for the right time to pursue your idea?

Whatever excuse you can come up with for why you’re waiting for the right time is probably not good enough. There is no such thing as “the right time”

If you are convinced about your idea. Stop what you are doing now, and go for it.

“goeie code” een mythe ?

Deze post gaat over een leuk dilemma in het ontwikkelen van projecten, meer bepaald het schrijven van code. Iedereen die al eens geprogrammeerd heeft aan een groter project (neen ik heb het niet over de myspace pagina van jouw neefje’s reggae band) kent het dilemma “goeie code” vs “snelle code”. Bij elk groter project komt er een bepaald moment waar je moet kiezen : de snelle (& dirty) oplossing of tijd investeren in de perfecte code ? Er zijn zeer weinig projecten waar de duurzame oplossing, maar vaak komt het hierop neer “if it works, it works”, niets is minder waar.

Elke die-hard developer weet dat de wereld beter zou worden van “goeie code”. Applicaties (in mijn geval web applicaties) zouden niet alleen doen wat ze horen te doen, ze zouden ook nog eens sneller zijn ! Het project process zou vlotter verlopen, en het eindresultaat zou voldoen aan alle eisen vooropgesteld in de analyse, right ? Spijtig genoeg is de realiteit anders.

Het eindresultaat is niet altijd afhankelijk van tijd/resources, er zijn ook andere factoren.

Ik heb in het verleden al aan tal van internet applicaties gewerkt, en altijd had ik de beste intenties voor de realisatie van het project. Altijd ! Beginnend met de analyse, alle actoren en processen zo goed mogelijk in kaart brengen en begrijpen hoe al die zaken uiteindelijk moeten samenwerken. Het resultaat was (meestal) duidelijke en gedocumenteerde code die gestructureerd, onderhoudbaar, leesbaar en uitbereidbaar was…tenminste toch tot de klant zich besloot te “moeien” :-)

Indien je al aan grotere projecten gewerkt hebt ken je zeker deze regel : “klanten weten nooit wat ze willen”. Zelfs als ze beweren van wel. Na het tonen van de eerste demo zijn ze verbaast en enthousiast zodat ze onmiddelijk idee’en beginnen te opperen om de applicatie “nog beter” en “nog cooler” te maken. Nu dit is niet echt een probleem zolang je de klant onder controle hebt (analyse document, project specificaties), maar dit niet het geval is : “let the bad code in” ! :-)

Op 90% van de reeds gerealiseerde projecten kwam er een moment waarop we keuzes moesten maken. Mogelijk kan er uit de eerste demo maar één voorstel van functionaliteit komen wat de klant geintegreerd wil zien. De kans is groot dat net deze functionaliteit zal botsen met alles wat je daarvoor al hebt geschreven. Daarom is het extreem moeilijk om “goeie code” te behouden. Het vereist oefening, ervaring en skill om dingen te schrijven die flexibel genoeg zijn om te voorzien aan de toenemende eisen van de klant.

Selah Sue

Das nog eens een belgisch export produkt !

[youtube]http://www.youtube.com/watch?v=h7Acu2Rm8vM[/youtube]

P.S. naar mijn mening zou Moby beter van de percussie afblijven !

Xhost/Domeincentrale : IVR

Op dit moment heeft wel elk IT bedrijfje een IVR, je kan zo snel met asterisk (of andere voip software) een telefooncentrale opzetten, en mensen die teweinig tijd (en geld teveel) hebben kunnen altijd eens bij belgacom aankloppen.

Ik had zonet iemand nodig van domeincentrale

Welcom bij Xhost group,
bent u klant van …. druk 1
bent u klant van domeincentrale….druk 2
bent u kla….
ik : 2

welkom bij domeincentrale
voor verkoop druk 1
voor technische support druk 2
ik 2–>tuut tuut, call ended…

Ok opnieuw :

Welcom bij Xhost group,
bent u klant van …. druk 1
bent u klant van domeincentrale….druk 2
bent u kla….
ik : 2

welkom bij domeincentrale
voor verkoop druk 1
voor technische support druk 2
ik 1

Welcom bij Xhost group,
bent u klant van …. druk 1
bent u klant van domeincentrale….druk 2
bent u kla….
ik : 2

GRRRRRRRRRRRRRRRRRRRRRRRRRR !!

&nbp;