Introduction

The project have three components: judger, backend, frontend, I am responsible for judger and backend

Mingdao Online Judge is my first website which took me over 200 hours to develop.

For the backend, we chose tonic, seaorm, and open-telemetry as the tech stack. On the other hand, we use actix-web, leptos, tonic-web for the front end. These choices are cutting-edge and allow us to enjoy the performance gains from low-level system programming languages.

However, the frontend hasn't been finished yet. My collaborator, Hugo, is currently working on that. Therefore, I would like to talk about the journey of making other part of Mingdao Online Judge.

Review of the progress

First, let's look at some commit message.

I have trimmed and fix typo on commit message.

Commit Message(You should skip it)

2023-12-23 18:39:16 +0800 eason chore: ✨ improve tracing
2023-12-20 00:40:21 +0800 eason refactor: :recycle: refactor judger
2023-12-18 20:03:48 +0800 kaiyohugo Merge branch 'master' of github.com:mdcpp/mdoj
2023-12-17 18:48:16 +0800 eason fix: 🐛 fix judger gcc plugin return unknown(CE should be return) during linker error
2023-12-17 18:47:09 +0800 eason fix: 🐛 fix judger gcc plugin return unknown(CE should be return) during linker error
2023-12-17 00:13:06 +0800 eason docs: 📝 change development setup to match present status
2023-12-12 23:24:05 +0800 eason chore: 🚀 update example
2023-12-12 19:13:13 +0800 eason feat: ✨ add ability to upload image
2023-12-11 13:31:32 +0800 eason refactor: ✨ add hard error for cgroupv1, use rustix instead of libc
2023-12-09 22:12:40 +0800 eason fix: 🐛 fix ProblemSet::list not utilize filter correctly
2023-12-09 20:14:28 +0800 eason change quickstart and setup config
2023-12-09 20:12:22 +0800 eason chore: ✨ setup working docker-compose
2023-12-09 17:27:44 +0800 eason feat: ✨ improve hit rate metrics
2023-12-09 00:52:57 +0800 eason chore: ✨ enable open telemetry by default
2023-12-08 23:36:19 +0800 eason perf: ⚡ parallelize server initialization
2023-12-08 00:26:37 +0800 eason test: :white_check_mark: able to connected to backend!
2023-12-08 00:01:56 +0800 eason refactor: 🐛 use fixed generated entity
2023-12-07 22:19:05 +0800 eason feat: ✨ draft migration
2023-12-07 20:55:51 +0800 eason init idens
2023-12-07 20:53:03 +0800 eason init idens
2023-12-07 15:58:24 +0800 eason test: 🧪 find cause of UB
2023-12-06 20:41:47 +0800 eason test: :construction: draft testsuit (test was a reserve keyword for rust)
2023-12-06 14:53:22 +0800 eason feat: ✨ add ability to perform passive healthcheck to judger, service discovery with built-in dns for docker swarm setup
2023-12-06 13:39:38 +0800 eason refactor: :construction: refactor router(thick client)
2023-12-05 13:14:00 +0800 eason refactor: :truck: rename submit controller to judger
2023-12-02 14:51:23 +0800 eason feat: :construction: draft playground set
2023-11-30 10:48:57 +0800 eason feat: ✨ improve judger memory protection, add exec rpc
2023-11-29 18:25:48 +0800 eason test: :white_check_mark: rlua-54 test pass
2023-11-27 00:34:40 +0800 eason style: :truck: cargo clippy
2023-11-26 15:16:00 +0800 eason docs: 📝 move dev.md to wiki
2023-11-24 23:08:01 +0800 eason feat: :white_check_mark: pass ass judger test
2023-11-23 20:17:25 +0800 eason feat: :construction: draft submit endpoint
2023-11-23 16:14:04 +0800 eason feat: 🐛 change judger proto, draft submit endpoints
2023-11-22 23:39:18 +0800 eason docs: 📝 update docs
2023-11-22 22:23:15 +0800 eason ci: ✨ finish ci
2023-11-22 22:22:12 +0800 eason add backend ci
2023-11-22 22:16:24 +0800 eason test ci
2023-11-22 22:15:54 +0800 eason fix sudo apt for ci
2023-11-22 22:07:23 +0800 eason test ci
2023-11-22 22:06:39 +0800 Eason0729 Update judger.yml
2023-11-22 22:01:31 +0800 eason Merge branch 'master' of github.com:mdcpp/mdoj
2023-11-22 22:01:16 +0800 eason test ci
2023-11-22 22:00:45 +0800 Eason0729 Create judger.yml
2023-11-22 21:46:57 +0800 eason docs: 📝 improve readme.md
2023-11-22 20:21:16 +0800 eason feat: ✨ finish image building script
2023-11-22 20:03:58 +0800 eason feat: ⚡ change nginx to nginx-slim
2023-11-22 20:02:29 +0800 eason feat: 🧪 add plugin gcc-13 and fail
2023-11-22 11:31:12 +0800 eason style: :recycle: cargo clippy
2023-11-22 11:22:31 +0800 eason refactor: :art: cargo clippy
2023-11-21 19:04:11 +0800 Eason0729 make nsjail musl and add working dockerfile (#9)
2023-11-21 13:26:25 +0800 eason feat: 🐛 update build method of nsjail
2023-11-21 00:50:43 +0800 eason feat: :construction: draft container compose
2023-11-20 20:25:45 +0800 eason feat: ✨ auto generate sqlite file
2023-11-19 23:05:50 +0800 eason fix: :white_check_mark: fix falling test
2023-11-19 16:23:48 +0800 eason docs: 📝 update documention for developer(hugo?)
2023-11-19 16:23:03 +0800 eason remove accidently commmited config file
2023-11-19 16:12:58 +0800 eason feat: :construction: auto db setup(smi-migration)
2023-11-19 15:55:32 +0800 eason feat: ✨ correct http2 grpc-web
2023-11-19 15:24:20 +0800 eason fix: ✨ fix grpc-web support
2023-11-18 21:44:36 +0800 eason refactor: :recycle: remove unused import, cargo fmt
2023-11-18 21:38:07 +0800 eason feat: ✨ basic auth support
2023-11-18 19:15:14 +0800 eason feat: :construction: fix config, judger connectivity
2023-10-25 23:20:27 +0800 eason fix(judger): 🚑 Fix nsjail not working not cgroupv2
2023-10-15 11:21:56 +0800 eason fix(judger): 🐛 OLE(don't resize vecter unintentially to correct memory claim)
2023-10-15 11:19:28 +0800 eason feat(judger): 🔧 rename, add configurable options
2023-10-11 20:37:06 +0800 eason test: 📌 add grpc test(note of unix socket)
2023-09-24 15:17:39 +0800 eason update readme.md for judger and judger's plugins
2023-09-17 15:34:22 +0800 eason finish editable trait
2023-09-17 12:12:40 +0800 eason add Create Endpoint
2023-09-16 15:19:01 +0800 eason move trait bound to lower ihneritance level implement select_only
2023-09-16 11:54:53 +0800 eason add intel.rs
2023-09-16 00:09:54 +0800 eason try templating list request
2023-09-15 10:24:13 +0800 eason finish proto, change entity
2023-09-04 10:45:05 +0800 eason add entity announcement, education
2023-08-29 10:36:02 +0800 eason define intel and edit trait
2023-08-22 13:23:46 +0800 eason pub/sub fix
2023-08-21 15:49:07 +0800 eason finish submit controller
2023-08-09 22:23:21 +0800 eason change database entity, draft TokenController
2023-08-09 16:42:23 +0800 eason finish database
2023-08-09 16:07:00 +0800 eason change layout
2023-08-09 16:06:52 +0800 eason update proto
2023-07-07 19:56:34 +0800 eason tested new container
2023-06-17 21:31:00 +0800 eason fix conflict test(prevent setting config twice)
2023-06-07 08:31:32 +0800 eason change sandbox config
2023-06-06 15:25:33 +0800 eason trim deps
2023-06-02 17:02:56 +0800 eason justfile and plugin build command
2023-06-02 16:47:07 +0800 eason change plugins detection
2023-06-02 13:56:58 +0800 eason tweak sandbox's error handling
2023-06-02 12:06:10 +0800 eason add health check for sandbox
2023-06-01 13:49:31 +0800 eason change sandbox behavior, new running status and OOM status
2023-05-31 15🔞30 +0800 eason generate entity
2023-05-31 10:41:34 +0800 eason init and first migration
2023-05-31 10:10:23 +0800 eason change deps of backend
2023-05-31 08:36:33 +0800 eason add resource preserve error
2023-05-30 14:52:51 +0800 eason add resource limit
2023-05-29 14:27:02 +0800 eason merge sandbox
2022-07-22 13:02:16 +0000 eason add .vscode config for debugger
2022-07-05 14:26:59 +0000 eason fix sql stat
2022-07-05 09:10:26 +0000 eason make entity standlone
2022-06-16 13:37:49 +0000 eason fmt
2022-06-16 13:37:05 +0000 eason token
2022-06-09 11:52:41 +0000 eason remove token
2022-06-08 14:45:36 +0000 eason token builder
2022-06-08 10:43:32 +0000 eason crypto test passed
2022-06-07 05:51:26 +0000 eason crypto test failed
2022-06-07 04:28:04 +0000 eason cache test passed
2022-06-07 04:25:57 +0000 eason cache test passed
2022-06-07 04:06:08 +0000 eason toml fix
2022-06-07 04:01:44 +0000 eason toml obstacle
2022-06-07 03:07:11 +0000 eason token::cache fix
2022-06-07 00:17:50 +0000 eason token::cache multithread test
2022-06-06 14:20:47 +0000 eason token::cache multithread failed
2022-06-06 11:49:06 +0000 eason token:refactor a part of code

In short, I have work on two crate, judger and backend.

The process can be illustrate as following:

1. Design new feature
2. Mock the future
3. Make the mock a reality
4. Write tests
5. Refactor (optional)
6. Back to (1)

Recently, I have reached a point where there are no further procedures to complete, so I have done a lot of refactoring. However, as it's close to the final exam period, I have to focus on preparing for it.

In total, I have spent over a hundred hours working on this project, and with each commit, I have learned and achieved so much. It's truly fascinating to realize how far I have come.

Inspiration

This project was heavily inspired by OnlineJudge. We decided to create MDOJ not only to enhance my programming skills but also to replace the original contest management system.

We collected complaints about the original system and choose several features that we believed MDOJ should implement, including:

A mechanism to reveal correct code (tutor of algorithm) to the user
An independent judger (to avoid TLE when the judger is congested)
Serverless judger

Challenges

This project has been a major milestone in my software development journey. I have never worked on such a large-scale project before, both in terms of performance expectations and technical complexity. By stepping out of my comfort zone, I have been able to expand my abilities even further.

Challenge of Low-level Languages

As comprehensive-rust course developed by the Google Android team highlights, Rust has following advantages(~~you can skip that~~), but those advantages come with some trade off.

advantage of Rust

High flexibility
High level of control
Can be scaled down to very constrained devices such as microcontrollers
Has no runtime or garbage collection
Focuses on reliability and safety without sacrificing performance

While this article is not about the cost of Rust, it's important to note that Rust has a steep learning curve and developing with it can be time-consuming. In our project, Rust has been used extensively.

Challenge of Sandboxing

One reason we decided to write our own contest management system is OnlineJudge's problem about sandboxing, their sandboxing is slow(start-up time) and reporting inaccurate resource usage.

Originally, we wanted to write our own sandboxing, and it works.. until we find it doesn't.

Correctly Sandboxing a application is really hard, hacker always have thousands of way to break it, so we ultimately choose to rely on a matured project to achieve it(we use nsjail).

Even I fail to build one our own, I have learn a lot about sandboxing on linux using some utility provided by linux kernel, so next time I have better chance to do one myself!

Challenge of scale

Perhaps you have worked on projects larger in scale than this one, but for me, it's the biggest one so far.

The most significant challenge when it comes to scale is the difficulty of maintainability. We definitely faced this challenge. For reference, I once wrote Rust code with five layers of indents, and at the time, I hadn't taken any object-oriented programming classes or similar courses. It was challenging to improve the structure and organization of the code.

To address this, I watched the video series "Computer Programming on NTU OpenCourseWare" (計算機程式設計- 臺大開放式課程), which helped me a lot. Although I still can't always write perfectly clean code, my collaborator seems able to understand most of my code. Additionally, the code is able to achieve an okay score (between 40-70 on the module level) using rust-code-analysis.

Overall, I'm satisfied with the progress made.

mdoj-contribute

How about next?

The backend refactoring is nearing completion. I'm eagerly anticipating a code review from an external expert, hoping that our code is in good quality.

Looking beyond, there's no immediate project on my schedule. The focus on academic performance and some rest are next, with lighter engagements like practicing LeetCode to temper the pace.

Thank you for reading my journey of developing MDOJ. If you are interested in contributing or doing code review for MDOJ? Email me or head over to our GitHub Repository!

Celebration of backend, judger finish!(mdoj)