Original article
https://senrigan.io/blog/how-writeupai-runs-behind-the-scenes/#h%5F6068056784021570782144062

Glossary

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

cross entropy loss
    (between two probability distributions)
    (and over the same underlying set of events)

    Higher loss is bad.

    Measures the performance of a
    classification model whose output is a
    probability value between 0 and 1.

    Measures the average number of bits needed
    to identify an event from the set.

Data scrubbing
    [error correction technique]

    Uses a background task to periodically
    inspect main memory or storage for errors,
    then correct detected errors using
    redundant data in the form of different
    checksums or copies of data.

openai/gradient-checkpointing
    [python/TensorFlow package]

    Handle GPU memory limitations.

Examples of usage of GPT-2

https://github.com/allenai/lm-explorer

Read this in eww

article review
GPT-2 Neural Network Poetry - Gwern.net http://mullikine.github.io/posts/review-of-gewn-gpt2/

The website when viewed with a GUI browser, although impressive, is distracting.

The website was built with hackyll, a static site generator employing haskell.

1

eww "https://www.gwern.net/GPT-2"

Straight-forward once you understand ML jargon.

Studying GPT-2

Preprocessing

1

cd "$DUMP$NOTES/ws/blog/posts"; wget 'http://static.decontextualize.com/gutenberg-poetry-v001.ndjson.gz'

ndjson

Each line corresponds to an independent json file.

Training

gradient checkpointing to handle memory issues

GPU memory is often the limiting factor for modern neural network architectures.

https://medium.com/tensorflow/fitting-larger-networks-into-memory-583e3c758ff9

Backend

https://github.com/jeffshek/open

$MYGIT/jeffshek/open

Glossary

1
2

Caddy
    HTTP/2 web server with automatic HTTPS.

Emacs setup

1

e ia caddyfile-mode

Caddy setup

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

:2015
 {
    proxy / django:8008 {
        header_upstream Host {host}
        header_upstream X-Real-IP {remote}
        header_upstream X-Forwarded-For {remote}
        header_upstream X-Forwarded-Proto {scheme}
        websocket
    }
    log stdout
    errors stdout
    tls off # tls/ssl then is turned off between the load balancer and web services for ssl termination
}

I guess the web server hosts django on =8008 and the caddy command below simple outputs errors to stdout.

When Caddyfile is present this should just work

1

caddy

Django setup

1

mkdir -p .envs/local && touch .django

Docker setup

1

docker-compose -f local.ym up