??
This commit is contained in:
parent
29aec69d4a
commit
10782342f2
111
out/blogs/nws-postmortem-11-8-23.html
Normal file
111
out/blogs/nws-postmortem-11-8-23.html
Normal file
|
|
@ -0,0 +1,111 @@
|
|||
<head>
|
||||
<title>Nicholas Orlowsky</title>
|
||||
<link rel="stylesheet" href="/style.css">
|
||||
<link rel="icon" type="image/x-icon" href="/favicon.ico">
|
||||
</head>
|
||||
<body>
|
||||
<nav>
|
||||
<a href="/">[ Home ]</a>
|
||||
<a href="/blog.html">[ Blog ]</a>
|
||||
<a href="/projects.html">[ Projects ]</a>
|
||||
<a href="/extra.html">[ Extra ]</a>
|
||||
<hr/>
|
||||
</nav>
|
||||
|
||||
<h1>NWS Incident Postmortem 11/08/2023</h1>
|
||||
|
||||
<p>
|
||||
On November 8th, 2023 at approximately 09:47 UTC, NWS suffered
|
||||
a complete outage. This outage resulted in the downtime of all
|
||||
services hosted on NWS and the downtime of the NWS Management
|
||||
Engine and the NWS dashboard.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
The incident lasted 28 minutes after which it was automatically
|
||||
resolved and all services were restored. This is NWS' first
|
||||
outage event of 2023.
|
||||
</p>
|
||||
|
||||
<h2>Cause</h2>
|
||||
<p>
|
||||
NWS utilizes several tactics to ensure uptime. A component of
|
||||
this is load balancing and failover. This service is currently
|
||||
provided by Cloudflare at the DNS level. Cloudflare sends
|
||||
health check requests to NWS servers at specified intervals. If
|
||||
it detects that one of the servers is down, it will remove the
|
||||
A record from entry.nws.nickorlow.com for that server (this domain
|
||||
is where all services on NWS direct their traffic via a
|
||||
CNAME).
|
||||
</p>
|
||||
|
||||
<p>
|
||||
At around 09:47 UTC, Cloudflare detected that our servers in
|
||||
Texas (Austin and Hill Country) were down. It did not detect an
|
||||
error, but rather an HTTP timeout. This is an indication that the
|
||||
server has lost network connectivity. When it detected that the
|
||||
servers were down, it removed their A records from the
|
||||
entry.nws.nickorlow.com domains. Since NWS' Pennsylvania servers
|
||||
have been undergoing maintenance since August 2023, this left no
|
||||
servers able to serve requests routed to entry.nws.nickorlow.com,
|
||||
resulting in the outage.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
NWS utilizes UptimeRobot for monitoring the uptime statistics of
|
||||
services on NWS and NWS servers. This is the source of the
|
||||
statistics shown on the NWS status page.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
UptimeRobot did not detect either of the Texas NWS servers as being
|
||||
offline for the duration of the outage. This is odd, as UptimeRobot
|
||||
and Cloudflare did not agree on the status of NWS servers. Logs
|
||||
on NWS servers showed that requests from UptimeRobot were being
|
||||
served while no requests from Cloudflare were shown in the logs.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
No firewall rules existed that could have blocked this traffic
|
||||
for either of the NWS servers. There was no other configuration
|
||||
found that would have blocked these requests. As these servers
|
||||
are on different networks inside different buildings in different
|
||||
parts of Texas, their networking equipment is entirely separate.
|
||||
This rules out any hardware failure of networking equipment owned
|
||||
by NWS. This leads us to believe that the issue may have been
|
||||
caused due to an internet traffic anomaly, although we are currently
|
||||
unable to confirm that this is the cause of the issue.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
This is being actively investigated to find a more concrete root
|
||||
cause. This postmortem will be updated if any new information is
|
||||
found.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
A similar event occurred on November 12th, 2023 lasting for 2 seconds.
|
||||
</p>
|
||||
|
||||
<h2>Fix</h2>
|
||||
<p>
|
||||
The common factor between both of these servers is that they both use
|
||||
Spectrum for their ISP and that they are located near Austin, Texas.
|
||||
The Pennsylvania server maintenance will be expedited so that we have
|
||||
servers online that operate with no commonalities.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
NWS will also investigate other methods of failover and load
|
||||
balancing.
|
||||
</p>
|
||||
|
||||
<p>Last updated on November 16th, 2023</p>
|
||||
|
||||
<footer>
|
||||
<hr />
|
||||
<p style="margin-bottom: 0px;">Copyright © Nicholas Orlowsky 2023</p>
|
||||
<p style="margin-top: 0px; margin-bottom: 0px;">Hosting provided by <a href="https://nws.nickorlow.com">NWS</a></p>
|
||||
<p style="margin-top: 0px;">Powered by <a href="https://github.com/nickorlow/anthracite">Anthracite Web Server</a></p>
|
||||
</footer>
|
||||
</body>
|
||||
121
out/blogs/side-project-10-20-23.html
Normal file
121
out/blogs/side-project-10-20-23.html
Normal file
|
|
@ -0,0 +1,121 @@
|
|||
<head>
|
||||
<title>Nicholas Orlowsky</title>
|
||||
<link rel="stylesheet" href="/style.css">
|
||||
<link rel="icon" type="image/x-icon" href="/favicon.ico">
|
||||
</head>
|
||||
<body>
|
||||
<nav>
|
||||
<a href="/">[ Home ]</a>
|
||||
<a href="/blog.html">[ Blog ]</a>
|
||||
<a href="/projects.html">[ Projects ]</a>
|
||||
<a href="/extra.html">[ Extra ]</a>
|
||||
<hr/>
|
||||
</nav>
|
||||
|
||||
<h1>Side Project Log 10/20/2023</h1>
|
||||
<p>This side project log covers work done from 8/15/2023 - 10/20/2023</p>
|
||||
|
||||
<h2 id="anthracite">Anthracite</h2>
|
||||
<a href="https://github.com/nickorlow/anthracite">[ GitHub Repo ]</a>
|
||||
<p>
|
||||
Anthracite is a web server written in C++. The site you're reading this on
|
||||
right now is hosted on Anthracite. I wrote it to deepen my knowledge of C++ and networking protocols. My
|
||||
main focus of Anthracite is performance. While developing anthracite,
|
||||
I have been exploring different optimization techniques and benchmarking
|
||||
Anthracite against popular web servers such as NGINX and Apache.
|
||||
Anthracite supports HTTP/1.1 and only supports GET requests to request
|
||||
files stored on a server.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
Anthracite currently performs on par with NGINX and Apache when making
|
||||
1000 requests for a 50MB file using 100 threads in a Docker container.
|
||||
To achieve this performance, I used memory profilers to find
|
||||
out what caused large or repeated memory copies to occur. I then updated
|
||||
those sections of code to remove or minimize these copies. I also
|
||||
made it so that Anthracite caches all files it can serve in memory. This
|
||||
avoids unnecessary and costly disk reads. The implementation of this is
|
||||
subpar, as it requires that the server be restarted whenever the files
|
||||
it is serving are changed for the updates to be detected by Anthracite.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
I intend to make further performance improvements, specifically in the request
|
||||
parser. I also plan to implement HTTP/2.0.
|
||||
</p>
|
||||
|
||||
<h2 id="yacemu">Yet Another Chip Eight Emulator (yacemu)</h2>
|
||||
<a href="https://github.com/nickorlow/yacemu">[ GitHub Repo ]</a>
|
||||
<p>
|
||||
YACEMU is an interpreter for the CHIP-8 instruction set written in C. My main
|
||||
goal when writing it was to gain more insight into how emulation works. I had
|
||||
previous experience with this from when I worked on an emulator for a slimmed-down
|
||||
version of X86 called <a href="https://web.cse.ohio-state.edu/~reeves.92/CSE2421sp13/PracticeProblemsY86.pdf">Y86</a>.
|
||||
So far, I've been able to get most instructions working. I need to work on adding
|
||||
input support so that users can interact with programs running in yacemu. It has
|
||||
been fairly uncomplicated and easy to write thus far. After I complete it, I would
|
||||
like to work on an emulator for a real device such as the GameBoy (This might be
|
||||
biting off more than I can chew).
|
||||
</p>
|
||||
|
||||
<h2 id="nick-vim">Nick VIM</h2>
|
||||
<p>
|
||||
Over the summer while I was interning, I began using VIM as my primary
|
||||
text editor. I used a preconfigured version of it (<a href="https://nvchad.com/">NvChad</a>) to save time, as
|
||||
setting everything up can take a while. After using it for a few months, I began
|
||||
making my own configuration for VIM, taking what I liked from NvChad and leaving
|
||||
behind the parts that I didn't like as much.
|
||||
</p>
|
||||
|
||||
<img src="/blog-images/NickVIM_Screenshot.png" alt="Screenshot of an HTML file open for editing in NickVIM"/>
|
||||
|
||||
<p>
|
||||
One important part of Nick VIM was ensuring that it was portable between different
|
||||
machines. I wanted the machine to have as few dependencies as possible so that I
|
||||
could get NickVIM set up on any computer in a couple of minutes. This will be especially
|
||||
useful when working on my School's lab machines and when switching to new computers
|
||||
in the future. I achieved this by dockerizing Nick VIM. This is based on what one of
|
||||
my co-workers does with their VIM setup. The Docker container contains
|
||||
all the dependencies for each language server. Whenever you edit a file with Nick Vim,
|
||||
the following script runs:
|
||||
</p>
|
||||
|
||||
<code lang="bash">
|
||||
echo Starting container...
|
||||
cur_dir=`pwd`
|
||||
container_name=${cur_dir////$'_'}
|
||||
container_name="${container_name:1}_$RANDOM"
|
||||
docker run --name $container_name --network host -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix --mount type=bind,source="$(pwd)",target=/work -d nick-vim &> /dev/null
|
||||
|
||||
echo Execing into container...
|
||||
docker exec -w /work -it $container_name bash
|
||||
|
||||
echo Stopping container in background...
|
||||
docker stop $container_name &> /dev/null &
|
||||
</code>
|
||||
|
||||
<p>
|
||||
This code creates a new container, forwards the host's clipboard to the container, and
|
||||
mounts the current directory inside the container for editing.
|
||||
</p>
|
||||
|
||||
<h2 id="secane">Secane</h2>
|
||||
<p><a href="https://www.youtube.com/watch?v=tKRehO7FH_s">[ Video Demo ]</a></p>
|
||||
<p>
|
||||
Secane was a simple ChatGPT wrapper that I wrote to practice for the behavioral part of
|
||||
job interviews. It takes your resume, information about the company, and information about
|
||||
the role you're interviewing for. It also integrates with OpenAI's whisper, allowing you
|
||||
to simulate talking out your answers. I made it with Next.JS.
|
||||
</p>
|
||||
|
||||
<hr/>
|
||||
<p><strong>These projects had minimal/no work done on them:</strong> NWS, RingGold, SQUIRREL</p>
|
||||
<p><strong>These projects I will no longer be working on:</strong> Olney</p>
|
||||
|
||||
<footer>
|
||||
<hr />
|
||||
<p style="margin-bottom: 0px;">Copyright © Nicholas Orlowsky 2023</p>
|
||||
<p style="margin-top: 0px; margin-bottom: 0px;">Hosting provided by <a href="https://nws.nickorlow.com">NWS</a></p>
|
||||
<p style="margin-top: 0px;">Powered by <a href="https://github.com/nickorlow/anthracite">Anthracite Web Server</a></p>
|
||||
</footer>
|
||||
</body>
|
||||
89
src/blogs/nws-postmortem-11-8-23.filler.html
Normal file
89
src/blogs/nws-postmortem-11-8-23.filler.html
Normal file
|
|
@ -0,0 +1,89 @@
|
|||
<h1>NWS Incident Postmortem 11/08/2023</h1>
|
||||
|
||||
<p>
|
||||
On November 8th, 2023 at approximately 09:47 UTC, NWS suffered
|
||||
a complete outage. This outage resulted in the downtime of all
|
||||
services hosted on NWS and the downtime of the NWS Management
|
||||
Engine and the NWS dashboard.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
The incident lasted 28 minutes after which it was automatically
|
||||
resolved and all services were restored. This is NWS' first
|
||||
outage event of 2023.
|
||||
</p>
|
||||
|
||||
<h2>Cause</h2>
|
||||
<p>
|
||||
NWS utilizes several tactics to ensure uptime. A component of
|
||||
this is load balancing and failover. This service is currently
|
||||
provided by Cloudflare at the DNS level. Cloudflare sends
|
||||
health check requests to NWS servers at specified intervals. If
|
||||
it detects that one of the servers is down, it will remove the
|
||||
A record from entry.nws.nickorlow.com for that server (this domain
|
||||
is where all services on NWS direct their traffic via a
|
||||
CNAME).
|
||||
</p>
|
||||
|
||||
<p>
|
||||
At around 09:47 UTC, Cloudflare detected that our servers in
|
||||
Texas (Austin and Hill Country) were down. It did not detect an
|
||||
error, but rather an HTTP timeout. This is an indication that the
|
||||
server has lost network connectivity. When it detected that the
|
||||
servers were down, it removed their A records from the
|
||||
entry.nws.nickorlow.com domains. Since NWS' Pennsylvania servers
|
||||
have been undergoing maintenance since August 2023, this left no
|
||||
servers able to serve requests routed to entry.nws.nickorlow.com,
|
||||
resulting in the outage.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
NWS utilizes UptimeRobot for monitoring the uptime statistics of
|
||||
services on NWS and NWS servers. This is the source of the
|
||||
statistics shown on the NWS status page.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
UptimeRobot did not detect either of the Texas NWS servers as being
|
||||
offline for the duration of the outage. This is odd, as UptimeRobot
|
||||
and Cloudflare did not agree on the status of NWS servers. Logs
|
||||
on NWS servers showed that requests from UptimeRobot were being
|
||||
served while no requests from Cloudflare were shown in the logs.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
No firewall rules existed that could have blocked this traffic
|
||||
for either of the NWS servers. There was no other configuration
|
||||
found that would have blocked these requests. As these servers
|
||||
are on different networks inside different buildings in different
|
||||
parts of Texas, their networking equipment is entirely separate.
|
||||
This rules out any hardware failure of networking equipment owned
|
||||
by NWS. This leads us to believe that the issue may have been
|
||||
caused due to an internet traffic anomaly, although we are currently
|
||||
unable to confirm that this is the cause of the issue.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
This is being actively investigated to find a more concrete root
|
||||
cause. This postmortem will be updated if any new information is
|
||||
found.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
A similar event occurred on November 12th, 2023 lasting for 2 seconds.
|
||||
</p>
|
||||
|
||||
<h2>Fix</h2>
|
||||
<p>
|
||||
The common factor between both of these servers is that they both use
|
||||
Spectrum for their ISP and that they are located near Austin, Texas.
|
||||
The Pennsylvania server maintenance will be expedited so that we have
|
||||
servers online that operate with no commonalities.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
NWS will also investigate other methods of failover and load
|
||||
balancing.
|
||||
</p>
|
||||
|
||||
<p>Last updated on November 16th, 2023</p>
|
||||
Loading…
Reference in a new issue