Merge branch 'or-references'
This commit is contained in:
commit
a705c9ae92
18 changed files with 749 additions and 2 deletions
|
|
@ -47,6 +47,10 @@
|
||||||
Here you can find all my previous posts:
|
Here you can find all my previous posts:
|
||||||
<ul>
|
<ul>
|
||||||
|
|
||||||
|
<li>
|
||||||
|
<a href="./posts/operations-research-references.html">Operations Research and Optimization: where to start?</a> - May 27, 2020
|
||||||
|
</li>
|
||||||
|
|
||||||
<li>
|
<li>
|
||||||
<a href="./posts/iclr-2020-notes.html">ICLR 2020 Notes: Speakers and Workshops</a> - May 5, 2020
|
<a href="./posts/iclr-2020-notes.html">ICLR 2020 Notes: Speakers and Workshops</a> - May 5, 2020
|
||||||
</li>
|
</li>
|
||||||
|
|
|
||||||
117
_site/atom.xml
117
_site/atom.xml
|
|
@ -8,8 +8,123 @@
|
||||||
<name>Dimitri Lozeve</name>
|
<name>Dimitri Lozeve</name>
|
||||||
<email>dimitri+web@lozeve.com</email>
|
<email>dimitri+web@lozeve.com</email>
|
||||||
</author>
|
</author>
|
||||||
<updated>2020-05-05T00:00:00Z</updated>
|
<updated>2020-05-27T00:00:00Z</updated>
|
||||||
<entry>
|
<entry>
|
||||||
|
<title>Operations Research and Optimization: where to start?</title>
|
||||||
|
<link href="https://www.lozeve.com/posts/operations-research-references.html" />
|
||||||
|
<id>https://www.lozeve.com/posts/operations-research-references.html</id>
|
||||||
|
<published>2020-05-27T00:00:00Z</published>
|
||||||
|
<updated>2020-05-27T00:00:00Z</updated>
|
||||||
|
<summary type="html"><![CDATA[<article>
|
||||||
|
<section class="header">
|
||||||
|
|
||||||
|
</section>
|
||||||
|
<section>
|
||||||
|
<p><a href="https://en.wikipedia.org/wiki/Operations_research">Operations research</a> (OR) is a vast area comprising a lot of theory, different branches of mathematics, and too many applications to count. In this post, I will try to explain why it can be a little disconcerting to explore at first, and how to start investigating the topic with a few references to get started.</p>
|
||||||
|
<p>Keep in mind that although I studied it during my graduate studies, this is not my primary area of expertise (I’m a data scientist by trade), and I definitely don’t pretend to know everything in OR. This is a field too vast for any single person to understand in its entirety, and I talk mostly from an “amateur mathematician and computer scientist” standpoint.</p>
|
||||||
|
<h2 id="why-is-it-hard-to-approach">Why is it hard to approach?</h2>
|
||||||
|
<p>Operations research can be difficult to approach, since there are many references and subfields. Compared to machine learning for instance, OR has a slightly longer history (going back to the 17th century, for example with <a href="https://en.wikipedia.org/wiki/Gaspard_Monge">Monge</a> and the <a href="https://en.wikipedia.org/wiki/Transportation_theory_(mathematics)">optimal transport problem</a>)<span><label for="sn-1" class="margin-toggle">⊕</label><input type="checkbox" id="sn-1" class="margin-toggle" /><span class="marginnote"> For a very nice introduction (in French) to optimal transport, see these blog posts by <a href="https://twitter.com/gabrielpeyre">Gabriel Peyré</a>, on the CNRS maths blog: <a href="https://images.math.cnrs.fr/Le-transport-optimal-numerique-et-ses-applications-Partie-1.html">Part 1</a> and <a href="https://images.math.cnrs.fr/Le-transport-optimal-numerique-et-ses-applications-Partie-2.html">Part 2</a>. See also the resources on <a href="https://optimaltransport.github.io/">optimaltransport.github.io</a> (in English).<br />
|
||||||
|
<br />
|
||||||
|
</span></span>. This means that good textbooks and such have existed for a long time, but also that there will be plenty of material to choose from.</p>
|
||||||
|
<p>Moreover, OR is very close to applications. Sometimes methods may vary a lot in their presentation depending on whether they’re applied to train tracks, sudoku, or travelling salesmen. In practice, the terminology and notations are not the same everywhere. This is disconcerting if you are used to “pure” mathematics, where notations evolved over a long time and is pretty much standardised for many areas. In contrast, if you’re used to the statistics literature with its <a href="https://lingpipe-blog.com/2009/10/13/whats-wrong-with-probability-notation/">strange notations</a>, you will find that OR is actually very well formalized.</p>
|
||||||
|
<p>There are many subfields of operations research, including all kinds of optimization (constrained and unconstrained), game theory, dynamic programming, stochastic processes, etc.</p>
|
||||||
|
<h2 id="where-to-start">Where to start</h2>
|
||||||
|
<h3 id="introduction-and-modelling">Introduction and modelling</h3>
|
||||||
|
<p>For an overall introduction, I recommend <span class="citation" data-cites="wentzel1988_operat">Wentzel (<a href="#ref-wentzel1988_operat">1988</a>)</span>. It is an old book, published by Mir Publications, a Soviet publisher which published many excellent scientific textbooks<span><label for="sn-2" class="margin-toggle">⊕</label><input type="checkbox" id="sn-2" class="margin-toggle" /><span class="marginnote"> Mir also published <a href="https://mirtitles.org/2011/06/03/physics-for-everyone/"><em>Physics for Everyone</em></a> by Lev Landau and Alexander Kitaigorodsky, a three-volume introduction to physics that is really accessible. Together with Feynman’s famous <a href="https://www.feynmanlectures.caltech.edu/">lectures</a>, I read them (in French) when I was a kid, and it was the best introduction I could possibly have to the subject.<br />
|
||||||
|
<br />
|
||||||
|
</span></span>. It is out of print, but it is available <a href="https://archive.org/details/WentzelOperationsResearchMir1983">on Archive.org</a><span><label for="sn-3" class="margin-toggle">⊕</label><input type="checkbox" id="sn-3" class="margin-toggle" /><span class="marginnote"><br />
|
||||||
|
<br />
|
||||||
|
<img src="/images/or_references/wentzel.jpg" width="200" /><br />
|
||||||
|
<br />
|
||||||
|
</span></span>. The book is quite old, but everything presented is still extremely relevant today. It requires absolutely no background, and covers everything: a general introduction to the field, linear programming, dynamic programming, Markov processes and queues, Monte Carlo methods, and game theory. Even if you already know some of these topics, the presentations is so clear that it is a pleasure to read! (In particular, it is one of the best presentations of dynamic programming that I have ever read. The explanation of the simplex algorithm is also excellent.)</p>
|
||||||
|
<p>If you are interested in optimization, the first thing you have to learn is modelling, i.e. transforming your problem (described in natural language, often from a particular industrial application) into a mathematical programme. The mathematical programme is the structure on which you will be able to apply an algorithm to find an optimal solution. Even if (like me) you are initially more interested in the algorithmic side of things, learning to create models will shed a lot of light on the overall process, and will give you more insight in general on the reasoning behind algorithms.</p>
|
||||||
|
<p>The best book I have read on the subject is <span class="citation" data-cites="williams2013_model">Williams (<a href="#ref-williams2013_model">2013</a>)</span><span><label for="sn-4" class="margin-toggle">⊕</label><input type="checkbox" id="sn-4" class="margin-toggle" /><span class="marginnote"><br />
|
||||||
|
<br />
|
||||||
|
<img src="/images/or_references/williams.jpg" width="200" /><br />
|
||||||
|
<br />
|
||||||
|
</span></span>. It contains a lot of concrete, step-by-step examples on concrete applications, in a multitude of domains, and remains very easy to read and to follow. It covers nearly every type of problem, so it is very useful as a reference. When you encounter a concrete problem in real life afterwards, you will know how to construct an appropriate model, and in the process you will often identify a common type of problem. The book then gives plenty of advice on how to approach each type of problem. Finally, it is also a great resource to build a “mental map” of the field, avoiding getting lost in the jungle of linear, stochastic, mixed integer, quadratic, and other network problems.</p>
|
||||||
|
<p>Another interesting resource is the freely available <a href="https://docs.mosek.com/modeling-cookbook/index.html">MOSEK Modeling Cookbook</a>, covering many types of problems, with more mathematical details than in <span class="citation" data-cites="williams2013_model">Williams (<a href="#ref-williams2013_model">2013</a>)</span>. It is built for people wanting to use the commercial <a href="https://www.mosek.com/">MOSEK</a> solver, so it could be useful if you plan to use a solver package like this one (more details on solvers <a href="#solvers">below</a>).</p>
|
||||||
|
<h3 id="theory-and-algorithms">Theory and algorithms</h3>
|
||||||
|
<p>The basic algorithm for optimization is the <a href="https://en.wikipedia.org/wiki/Simplex_algorithm">simplex algorithm</a>, developed by Dantzig in the 1940s to solve <a href="https://en.wikipedia.org/wiki/Linear_programming">linear programming</a> problems. It is the one of the main building blocks for mathematical optimization, and is used and referenced extensively in all kinds of approaches. As such, it is really important to understand it in detail. There are many books on the subject, but I especially liked <span class="citation" data-cites="chvatal1983_linear">Chvátal (<a href="#ref-chvatal1983_linear">1983</a>)</span> (out of print, but you can find cheap used versions on Amazon). It covers everything there is to know on the simplex algorithms (step-by-step explanations with simple examples, correctness and complexity analysis, computational and implementation considerations) and to many applications. I think it is overall the best introduction. <span class="citation" data-cites="vanderbei2014_linear">Vanderbei (<a href="#ref-vanderbei2014_linear">2014</a>)</span> follows a very similar outline, but contains more recent computational considerations<span><label for="sn-5" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-5" class="margin-toggle" /><span class="sidenote">For all the details about practical implementations of the simplex algorithm, <span class="citation" data-cites="maros2003_comput">Maros (<a href="#ref-maros2003_comput">2003</a>)</span> is dedicated to the computational aspects and contains everything you will need.<br />
|
||||||
|
<br />
|
||||||
|
</span></span>. (The author also has <a href="http://vanderbei.princeton.edu/307/lectures.html">lecture slides</a>.)</p>
|
||||||
|
<p>For more books on linear programming, the two books <span class="citation" data-cites="dantzig1997_linear">Dantzig (<a href="#ref-dantzig1997_linear">1997</a>)</span>, <span class="citation" data-cites="dantzig2003_linear">Dantzig (<a href="#ref-dantzig2003_linear">2003</a>)</span> are very complete, if somewhat more mathematically advanced. <span class="citation" data-cites="bertsimas1997_introd">Bertsimas and Tsitsiklis (<a href="#ref-bertsimas1997_introd">1997</a>)</span> is also a great reference, if you can find it.</p>
|
||||||
|
<p>For all the other subfields, <a href="https://or.stackexchange.com/a/870">this great StackExchange answer</a> contains a lot of useful references, including most of the above. Of particular note are <span class="citation" data-cites="peyreComputationalOptimalTransport2019">Peyré and Cuturi (<a href="#ref-peyreComputationalOptimalTransport2019">2019</a>)</span> for optimal transport, <span class="citation" data-cites="boyd2004_convex">Boyd (<a href="#ref-boyd2004_convex">2004</a>)</span> for convex optimization (<a href="https://web.stanford.edu/~boyd/cvxbook/">freely available online</a>), and <span class="citation" data-cites="nocedal2006_numer">Nocedal (<a href="#ref-nocedal2006_numer">2006</a>)</span> for numerical optimization. <span class="citation" data-cites="kochenderfer2019_algor">Kochenderfer (<a href="#ref-kochenderfer2019_algor">2019</a>)</span><span><label for="sn-6" class="margin-toggle">⊕</label><input type="checkbox" id="sn-6" class="margin-toggle" /><span class="marginnote"><br />
|
||||||
|
<br />
|
||||||
|
<img src="/images/or_references/kochenderfer.jpg" width="200" /><br />
|
||||||
|
<br />
|
||||||
|
</span></span> is not in the list (because it is very recent) but is also excellent, with examples in Julia covering nearly every kind of optimization algorithms.</p>
|
||||||
|
<h3 id="online-courses">Online courses</h3>
|
||||||
|
<p>If you would like to watch video lectures, there are a few good opportunities freely available online, in particular on <a href="https://ocw.mit.edu/index.htm">MIT OpenCourseWare</a>. The list of courses at MIT is available <a href="https://orc.mit.edu/academics/course-offerings">on their webpage</a>. I haven’t actually looked in details at the courses content<span><label for="sn-7" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-7" class="margin-toggle" /><span class="sidenote">I am more comfortable reading books than watching lecture videos online. Although I liked attending classes during my studies, I do not have the same feeling in front of a video. When I read, I can re-read three times the same sentence, pause to look up something, or skim a few paragraphs. I find that the inability to do that with a video diminishes greatly my ability to concentrate.<br />
|
||||||
|
<br />
|
||||||
|
</span></span>, so I cannot vouch for them directly, but MIT courses are generally of excellent quality. Most courses are also taught by Bertsimas and Bertsekas, who are very famous and wrote many excellent books.</p>
|
||||||
|
<p>Of particular notes are:</p>
|
||||||
|
<ul>
|
||||||
|
<li><a href="https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-251j-introduction-to-mathematical-programming-fall-2009/">Introduction to Mathematical Programming</a>,</li>
|
||||||
|
<li><a href="https://ocw.mit.edu/courses/sloan-school-of-management/15-084j-nonlinear-programming-spring-2004/">Nonlinear Optimization</a>,</li>
|
||||||
|
<li><a href="https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-253-convex-analysis-and-optimization-spring-2012/">Convex Analysis and Optimization</a>,</li>
|
||||||
|
<li><a href="https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-972-algebraic-techniques-and-semidefinite-optimization-spring-2006/">Algebraic Techniques and Semidefinite Optimization</a>,</li>
|
||||||
|
<li><a href="https://ocw.mit.edu/courses/sloan-school-of-management/15-083j-integer-programming-and-combinatorial-optimization-fall-2009/">Integer Programming and Combinatorial Optimization</a>.</li>
|
||||||
|
</ul>
|
||||||
|
<p>Another interesting course I found online is <a href="https://www.ams.jhu.edu/~wcook12/dl/index.html">Deep Learning in Discrete Optimization</a>, at Johns Hopkins<span><label for="sn-8" class="margin-toggle">⊕</label><input type="checkbox" id="sn-8" class="margin-toggle" /><span class="marginnote"> It is taught by William Cook, who is the author of <a href="https://press.princeton.edu/books/paperback/9780691163529/in-pursuit-of-the-traveling-salesman"><em>In Pursuit of the Traveling Salesman</em></a>, a nice introduction to the TSP problem in a readable form.<br />
|
||||||
|
<br />
|
||||||
|
</span></span>. It contains an interesting overview of deep learning and integer programming, with a focus on connections, and applications to recent research areas in ML (reinforcement learning, attention, etc.).</p>
|
||||||
|
<h2 id="solvers-and-computational-resources">Solvers and computational resources <span id="solvers"></span></h2>
|
||||||
|
<p>When you start reading about modelling and algorithms, I recommend you try solving a few problems yourself, either by hand for small instances, or using an existing solver. It will allow you to follow the examples in books, while also practising your modelling skills. You will also get an intuition of what is difficult to model and to solve.</p>
|
||||||
|
<p>There are many solvers available, both free and commercial, with various capabilities. I recommend you use the fantastic <a href="https://github.com/JuliaOpt/JuMP.jl">JuMP</a><span><label for="sn-9" class="margin-toggle">⊕</label><input type="checkbox" id="sn-9" class="margin-toggle" /><span class="marginnote"><br />
|
||||||
|
<br />
|
||||||
|
<img src="/images/or_references/jump.svg" width="250" /><br />
|
||||||
|
<br />
|
||||||
|
</span></span> library for Julia, which exposes a domain-specific language for modelling, along with interfaces to nearly all major solver packages. (Even if you don’t know Julia, this is a great and easy way to start!) If you’d rather use Python, you can use Google’s <a href="https://developers.google.com/optimization/introduction/python">OR-Tools</a> or <a href="https://github.com/coin-or/pulp">PuLP</a> for linear programming.</p>
|
||||||
|
<p>Regarding solvers, there is a <a href="http://www.juliaopt.org/JuMP.jl/stable/installation/#Getting-Solvers-1">list of solvers</a> on JuMP’s documentation, with their capabilities and their license. Free solvers include <a href="https://www.gnu.org/software/glpk/">GLPK</a> (linear programming), <a href="https://github.com/coin-or/Ipopt">Ipopt</a> (non-linear programming), and <a href="https://scip.zib.de/">SCIP</a> (mixed-integer linear programming).</p>
|
||||||
|
<p>Commercial solvers often have better performance, and some of them propose a free academic license: <a href="https://www.mosek.com/">MOSEK</a>, <a href="https://www.gurobi.com/">Gurobi</a>, and <a href="https://www.ibm.com/analytics/cplex-optimizer">IBM CPLEX</a> in particular all offer free academic licenses and work very well with JuMP.</p>
|
||||||
|
<p>Another awesome resource is the <a href="https://neos-server.org/neos/">NEOS Server</a>. It offers free computing resources for numerical optimization, including all major free and commercial solvers! You can submit jobs on it in a standard format, or interface your favourite programming language with it. The fact that such an amazing resource exists for free, for everyone is extraordinary. They also have an accompanying book, the <a href="https://neos-guide.org/">NEOS Guide</a>, containing many case studies and description of problem types. The <a href="https://neos-guide.org/content/optimization-taxonomy">taxonomy</a> may be particularly useful.</p>
|
||||||
|
<h2 id="conclusion">Conclusion</h2>
|
||||||
|
<p>Operations research is a fascinating topic, and it has an abundant literature that makes it very easy to dive into the subject. If you are interested in algorithms, modelling for practical applications, or just wish to understand more, I hope to have given you the first steps to follow, start reading and experimenting.</p>
|
||||||
|
<h2 id="references" class="unnumbered">References</h2>
|
||||||
|
<div id="refs" class="references">
|
||||||
|
<div id="ref-bertsimas1997_introd">
|
||||||
|
<p>Bertsimas, Dimitris, and John N. Tsitsiklis. 1997. <em>Introduction to Linear Optimization</em>. Belmont, Massachusetts: Athena Scientific. <a href="http://www.athenasc.com/linoptbook.html" class="uri">http://www.athenasc.com/linoptbook.html</a>.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-boyd2004_convex">
|
||||||
|
<p>Boyd, Stephen. 2004. <em>Convex Optimization</em>. Cambridge, UK New York: Cambridge University Press.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-chvatal1983_linear">
|
||||||
|
<p>Chvátal, Vašek. 1983. <em>Linear Programming</em>. New York: W.H. Freeman.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-dantzig1997_linear">
|
||||||
|
<p>Dantzig, George. 1997. <em>Linear Programming 1: Introduction</em>. New York: Springer. <a href="https://www.springer.com/gp/book/9780387948331" class="uri">https://www.springer.com/gp/book/9780387948331</a>.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-dantzig2003_linear">
|
||||||
|
<p>———. 2003. <em>Linear Programming 2: Theory and Extensions</em>. New York: Springer. <a href="https://www.springer.com/gp/book/9780387986135" class="uri">https://www.springer.com/gp/book/9780387986135</a>.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-kochenderfer2019_algor">
|
||||||
|
<p>Kochenderfer, Mykel. 2019. <em>Algorithms for Optimization</em>. Cambridge, Massachusetts: The MIT Press.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-maros2003_comput">
|
||||||
|
<p>Maros, István. 2003. <em>Computational Techniques of the Simplex Method</em>. Boston: Kluwer Academic Publishers.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-nocedal2006_numer">
|
||||||
|
<p>Nocedal, Jorge. 2006. <em>Numerical Optimization</em>. New York: Springer. <a href="https://www.springer.com/gp/book/9780387303031" class="uri">https://www.springer.com/gp/book/9780387303031</a>.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-peyreComputationalOptimalTransport2019">
|
||||||
|
<p>Peyré, Gabriel, and Marco Cuturi. 2019. “Computational Optimal Transport.” <em>Foundations and Trends in Machine Learning</em> 11 (5-6): 355–206. <a href="https://doi.org/10.1561/2200000073" class="uri">https://doi.org/10.1561/2200000073</a>.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-vanderbei2014_linear">
|
||||||
|
<p>Vanderbei, Robert. 2014. <em>Linear Programming : Foundations and Extensions</em>. New York: Springer.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-wentzel1988_operat">
|
||||||
|
<p>Wentzel, Elena S. 1988. <em>Operations Research: A Methodological Approach</em>. Moscow: Mir publishers.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-williams2013_model">
|
||||||
|
<p>Williams, H. Paul. 2013. <em>Model Building in Mathematical Programming</em>. Chichester, West Sussex: Wiley. <a href="https://www.wiley.com/en-fr/Model+Building+in+Mathematical+Programming,+5th+Edition-p-9781118443330" class="uri">https://www.wiley.com/en-fr/Model+Building+in+Mathematical+Programming,+5th+Edition-p-9781118443330</a>.</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
</article>
|
||||||
|
]]></summary>
|
||||||
|
</entry>
|
||||||
|
<entry>
|
||||||
<title>ICLR 2020 Notes: Speakers and Workshops</title>
|
<title>ICLR 2020 Notes: Speakers and Workshops</title>
|
||||||
<link href="https://www.lozeve.com/posts/iclr-2020-notes.html" />
|
<link href="https://www.lozeve.com/posts/iclr-2020-notes.html" />
|
||||||
<id>https://www.lozeve.com/posts/iclr-2020-notes.html</id>
|
<id>https://www.lozeve.com/posts/iclr-2020-notes.html</id>
|
||||||
|
|
|
||||||
1
_site/images/or_references/jump.svg
Normal file
1
_site/images/or_references/jump.svg
Normal file
|
|
@ -0,0 +1 @@
|
||||||
|
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 377.42 122.03"><defs><style>.cls-1{fill:#231f20;}.cls-2,.cls-3,.cls-5,.cls-7,.cls-9{fill:none;}.cls-2,.cls-3{stroke:#231f20;stroke-linecap:round;stroke-linejoin:round;}.cls-2{stroke-width:6.63px;}.cls-3{stroke-width:6.63px;fill-rule:evenodd;}.cls-4{fill:#aa7dc0;}.cls-5{stroke:#945bb0;}.cls-5,.cls-7,.cls-9{stroke-miterlimit:10;stroke-width:3px;}.cls-6{fill:#6bab5b;}.cls-7{stroke:#3b972e;}.cls-8{fill:#d66661;}.cls-9{stroke:#c93d39;}</style></defs><title>Thin-Border-Logo-Text</title><g id="Page_1" data-name="Page 1"><g id="Layer_1-2" data-name="Layer 1"><polygon class="cls-1" points="313.71 117.21 326.3 117.21 326.3 50.53 311.9 50.53 291.01 79.2 270.12 50.53 255.62 50.53 255.62 117.21 268.21 117.21 268.21 68.72 289.48 97.78 291.96 97.78 313.71 68.81 313.71 117.21 313.71 117.21"/><path class="cls-1" d="M172.45,50.54H185.2V93.37c0,3.86.29,7.71-.65,11.5a19.26,19.26,0,0,1-7.72,11.33,19.75,19.75,0,0,1-9.07,3.42,20.85,20.85,0,0,1-22.67-16.11l12.38-3.16a8.45,8.45,0,0,0,5.17,6.09,8,8,0,0,0,3.67.45,7.09,7.09,0,0,0,3.2-1.16c2.77-1.93,2.94-5.34,2.94-8.34V50.54Zm26.46,0h12.52V74.35c0,4.61.31,7.84.95,9.63a8.46,8.46,0,0,0,3.06,4.19,8.76,8.76,0,0,0,5.2,1.51,9.15,9.15,0,0,0,5.24-1.47,8.68,8.68,0,0,0,3.2-4.34q.76-2.13.76-9.12V50.57h12.43V71.49c0,8.61-.69,14.52-2,17.69a19.21,19.21,0,0,1-7.34,8.9q-4.86,3.11-12.34,3.11-8.13,0-13.13-3.62a19.27,19.27,0,0,1-7.06-10.12c-.94-3-1.45-8.44-1.45-16.32V50.57Zm138.87,0h13.46q11,0,15.76,2a16,16,0,0,1,7.63,6.54,20.21,20.21,0,0,1,2.79,10.8,18.23,18.23,0,0,1-3.65,11.61A18.82,18.82,0,0,1,363.86,88q-3.66,1-13.37,1v28.26H337.78V50.54Zm12.71,26.08h4a22.42,22.42,0,0,0,6.63-.68,5.82,5.82,0,0,0,2.92-2.25,6.53,6.53,0,0,0,1.07-3.79,6.09,6.09,0,0,0-3-5.62C360.69,63.41,358,63,354.06,63h-3.57V76.62Z"/><line class="cls-2" x1="19.64" y1="67.28" x2="120.16" y2="28.42"/><polyline class="cls-3" points="3.32 118.72 89.75 32.29 118.72 3.32"/><path class="cls-4" d="M106.29,120.18a17.17,17.17,0,1,1,12.16-5A17.06,17.06,0,0,1,106.29,120.18Z"/><path class="cls-5" d="M106.29,120.18a17.17,17.17,0,1,1,12.16-5A17.06,17.06,0,0,1,106.29,120.18Z"/><path class="cls-6" d="M106.29,79.33a17.19,17.19,0,1,1,12.16-5A17.07,17.07,0,0,1,106.29,79.33Z"/><path class="cls-7" d="M106.29,79.33a17.19,17.19,0,1,1,12.16-5A17.07,17.07,0,0,1,106.29,79.33Z"/><path class="cls-8" d="M65.28,120.18a17.19,17.19,0,1,1,12.15-5A17,17,0,0,1,65.28,120.18Z"/><path class="cls-9" d="M65.28,120.18a17.19,17.19,0,1,1,12.15-5A17,17,0,0,1,65.28,120.18Z"/></g></g></svg>
|
||||||
|
After Width: | Height: | Size: 2.5 KiB |
BIN
_site/images/or_references/kochenderfer.jpg
Normal file
BIN
_site/images/or_references/kochenderfer.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 54 KiB |
BIN
_site/images/or_references/kochenderfer_small.jpg
Normal file
BIN
_site/images/or_references/kochenderfer_small.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 54 KiB |
BIN
_site/images/or_references/wentzel.jpg
Normal file
BIN
_site/images/or_references/wentzel.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 108 KiB |
BIN
_site/images/or_references/wentzel_small.jpg
Normal file
BIN
_site/images/or_references/wentzel_small.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 108 KiB |
BIN
_site/images/or_references/williams.jpg
Normal file
BIN
_site/images/or_references/williams.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 52 KiB |
BIN
_site/images/or_references/williams_small.jpg
Normal file
BIN
_site/images/or_references/williams_small.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 52 KiB |
|
|
@ -70,6 +70,10 @@ public key: RWQ6uexORp8f7USHA7nX9lFfltaCA9x6aBV06MvgiGjUt6BVf6McyD26
|
||||||
|
|
||||||
<ul>
|
<ul>
|
||||||
|
|
||||||
|
<li>
|
||||||
|
<a href="./posts/operations-research-references.html">Operations Research and Optimization: where to start?</a> - May 27, 2020
|
||||||
|
</li>
|
||||||
|
|
||||||
<li>
|
<li>
|
||||||
<a href="./posts/iclr-2020-notes.html">ICLR 2020 Notes: Speakers and Workshops</a> - May 5, 2020
|
<a href="./posts/iclr-2020-notes.html">ICLR 2020 Notes: Speakers and Workshops</a> - May 5, 2020
|
||||||
</li>
|
</li>
|
||||||
|
|
|
||||||
161
_site/posts/operations-research-references.html
Normal file
161
_site/posts/operations-research-references.html
Normal file
|
|
@ -0,0 +1,161 @@
|
||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<meta http-equiv="x-ua-compatible" content="ie=edge">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=yes">
|
||||||
|
<meta name="description" content="Dimitri Lozeve's blog: Operations Research and Optimization: where to start?">
|
||||||
|
|
||||||
|
<title>Dimitri Lozeve - Operations Research and Optimization: where to start?</title>
|
||||||
|
<link rel="stylesheet" href="../css/tufte.css" />
|
||||||
|
<link rel="stylesheet" href="../css/pandoc.css" />
|
||||||
|
<link rel="stylesheet" href="../css/default.css" />
|
||||||
|
<link rel="stylesheet" href="../css/syntax.css" />
|
||||||
|
|
||||||
|
<!-- KaTeX CSS styles -->
|
||||||
|
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.11.0/dist/katex.min.css" integrity="sha384-BdGj8xC2eZkQaxoQ8nSLefg4AV4/AwB3Fj+8SUSo7pnKP6Eoy18liIKTPn9oBYNG" crossorigin="anonymous">
|
||||||
|
|
||||||
|
<!-- The loading of KaTeX is deferred to speed up page rendering -->
|
||||||
|
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.11.0/dist/katex.min.js" integrity="sha384-JiKN5O8x9Hhs/UE5cT5AAJqieYlOZbGT3CHws/y97o3ty4R7/O5poG9F3JoiOYw1" crossorigin="anonymous"></script>
|
||||||
|
|
||||||
|
<!-- To automatically render math in text elements, include the auto-render extension: -->
|
||||||
|
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.11.0/dist/contrib/auto-render.min.js" integrity="sha384-kWPLUVMOks5AQFrykwIup5lo0m3iMkkHrD0uJ4H5cjeGihAutqP0yW0J6dpFiVkI" crossorigin="anonymous" onload="renderMathInElement(document.body);"></script>
|
||||||
|
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<article>
|
||||||
|
|
||||||
|
<header>
|
||||||
|
<nav>
|
||||||
|
<a href="../">Home</a>
|
||||||
|
<a href="../projects.html">Projects</a>
|
||||||
|
<a href="../archive.html">Archive</a>
|
||||||
|
<a href="../contact.html">Contact</a>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
<h1 class="title">Operations Research and Optimization: where to start?</h1>
|
||||||
|
|
||||||
|
|
||||||
|
<p class="byline">May 27, 2020</p>
|
||||||
|
|
||||||
|
</header>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
</article>
|
||||||
|
|
||||||
|
<article>
|
||||||
|
<section class="header">
|
||||||
|
|
||||||
|
</section>
|
||||||
|
<section>
|
||||||
|
<p><a href="https://en.wikipedia.org/wiki/Operations_research">Operations research</a> (OR) is a vast area comprising a lot of theory, different branches of mathematics, and too many applications to count. In this post, I will try to explain why it can be a little disconcerting to explore at first, and how to start investigating the topic with a few references to get started.</p>
|
||||||
|
<p>Keep in mind that although I studied it during my graduate studies, this is not my primary area of expertise (I’m a data scientist by trade), and I definitely don’t pretend to know everything in OR. This is a field too vast for any single person to understand in its entirety, and I talk mostly from an “amateur mathematician and computer scientist” standpoint.</p>
|
||||||
|
<h2 id="why-is-it-hard-to-approach">Why is it hard to approach?</h2>
|
||||||
|
<p>Operations research can be difficult to approach, since there are many references and subfields. Compared to machine learning for instance, OR has a slightly longer history (going back to the 17th century, for example with <a href="https://en.wikipedia.org/wiki/Gaspard_Monge">Monge</a> and the <a href="https://en.wikipedia.org/wiki/Transportation_theory_(mathematics)">optimal transport problem</a>)<span><label for="sn-1" class="margin-toggle">⊕</label><input type="checkbox" id="sn-1" class="margin-toggle" /><span class="marginnote"> For a very nice introduction (in French) to optimal transport, see these blog posts by <a href="https://twitter.com/gabrielpeyre">Gabriel Peyré</a>, on the CNRS maths blog: <a href="https://images.math.cnrs.fr/Le-transport-optimal-numerique-et-ses-applications-Partie-1.html">Part 1</a> and <a href="https://images.math.cnrs.fr/Le-transport-optimal-numerique-et-ses-applications-Partie-2.html">Part 2</a>. See also the resources on <a href="https://optimaltransport.github.io/">optimaltransport.github.io</a> (in English).<br />
|
||||||
|
<br />
|
||||||
|
</span></span>. This means that good textbooks and such have existed for a long time, but also that there will be plenty of material to choose from.</p>
|
||||||
|
<p>Moreover, OR is very close to applications. Sometimes methods may vary a lot in their presentation depending on whether they’re applied to train tracks, sudoku, or travelling salesmen. In practice, the terminology and notations are not the same everywhere. This is disconcerting if you are used to “pure” mathematics, where notations evolved over a long time and is pretty much standardised for many areas. In contrast, if you’re used to the statistics literature with its <a href="https://lingpipe-blog.com/2009/10/13/whats-wrong-with-probability-notation/">strange notations</a>, you will find that OR is actually very well formalized.</p>
|
||||||
|
<p>There are many subfields of operations research, including all kinds of optimization (constrained and unconstrained), game theory, dynamic programming, stochastic processes, etc.</p>
|
||||||
|
<h2 id="where-to-start">Where to start</h2>
|
||||||
|
<h3 id="introduction-and-modelling">Introduction and modelling</h3>
|
||||||
|
<p>For an overall introduction, I recommend <span class="citation" data-cites="wentzel1988_operat">Wentzel (<a href="#ref-wentzel1988_operat">1988</a>)</span>. It is an old book, published by Mir Publications, a Soviet publisher which published many excellent scientific textbooks<span><label for="sn-2" class="margin-toggle">⊕</label><input type="checkbox" id="sn-2" class="margin-toggle" /><span class="marginnote"> Mir also published <a href="https://mirtitles.org/2011/06/03/physics-for-everyone/"><em>Physics for Everyone</em></a> by Lev Landau and Alexander Kitaigorodsky, a three-volume introduction to physics that is really accessible. Together with Feynman’s famous <a href="https://www.feynmanlectures.caltech.edu/">lectures</a>, I read them (in French) when I was a kid, and it was the best introduction I could possibly have to the subject.<br />
|
||||||
|
<br />
|
||||||
|
</span></span>. It is out of print, but it is available <a href="https://archive.org/details/WentzelOperationsResearchMir1983">on Archive.org</a><span><label for="sn-3" class="margin-toggle">⊕</label><input type="checkbox" id="sn-3" class="margin-toggle" /><span class="marginnote"><br />
|
||||||
|
<br />
|
||||||
|
<img src="../images/or_references/wentzel.jpg" width="200" /><br />
|
||||||
|
<br />
|
||||||
|
</span></span>. The book is quite old, but everything presented is still extremely relevant today. It requires absolutely no background, and covers everything: a general introduction to the field, linear programming, dynamic programming, Markov processes and queues, Monte Carlo methods, and game theory. Even if you already know some of these topics, the presentations is so clear that it is a pleasure to read! (In particular, it is one of the best presentations of dynamic programming that I have ever read. The explanation of the simplex algorithm is also excellent.)</p>
|
||||||
|
<p>If you are interested in optimization, the first thing you have to learn is modelling, i.e. transforming your problem (described in natural language, often from a particular industrial application) into a mathematical programme. The mathematical programme is the structure on which you will be able to apply an algorithm to find an optimal solution. Even if (like me) you are initially more interested in the algorithmic side of things, learning to create models will shed a lot of light on the overall process, and will give you more insight in general on the reasoning behind algorithms.</p>
|
||||||
|
<p>The best book I have read on the subject is <span class="citation" data-cites="williams2013_model">Williams (<a href="#ref-williams2013_model">2013</a>)</span><span><label for="sn-4" class="margin-toggle">⊕</label><input type="checkbox" id="sn-4" class="margin-toggle" /><span class="marginnote"><br />
|
||||||
|
<br />
|
||||||
|
<img src="../images/or_references/williams.jpg" width="200" /><br />
|
||||||
|
<br />
|
||||||
|
</span></span>. It contains a lot of concrete, step-by-step examples on concrete applications, in a multitude of domains, and remains very easy to read and to follow. It covers nearly every type of problem, so it is very useful as a reference. When you encounter a concrete problem in real life afterwards, you will know how to construct an appropriate model, and in the process you will often identify a common type of problem. The book then gives plenty of advice on how to approach each type of problem. Finally, it is also a great resource to build a “mental map” of the field, avoiding getting lost in the jungle of linear, stochastic, mixed integer, quadratic, and other network problems.</p>
|
||||||
|
<p>Another interesting resource is the freely available <a href="https://docs.mosek.com/modeling-cookbook/index.html">MOSEK Modeling Cookbook</a>, covering many types of problems, with more mathematical details than in <span class="citation" data-cites="williams2013_model">Williams (<a href="#ref-williams2013_model">2013</a>)</span>. It is built for people wanting to use the commercial <a href="https://www.mosek.com/">MOSEK</a> solver, so it could be useful if you plan to use a solver package like this one (more details on solvers <a href="#solvers">below</a>).</p>
|
||||||
|
<h3 id="theory-and-algorithms">Theory and algorithms</h3>
|
||||||
|
<p>The basic algorithm for optimization is the <a href="https://en.wikipedia.org/wiki/Simplex_algorithm">simplex algorithm</a>, developed by Dantzig in the 1940s to solve <a href="https://en.wikipedia.org/wiki/Linear_programming">linear programming</a> problems. It is the one of the main building blocks for mathematical optimization, and is used and referenced extensively in all kinds of approaches. As such, it is really important to understand it in detail. There are many books on the subject, but I especially liked <span class="citation" data-cites="chvatal1983_linear">Chvátal (<a href="#ref-chvatal1983_linear">1983</a>)</span> (out of print, but you can find cheap used versions on Amazon). It covers everything there is to know on the simplex algorithms (step-by-step explanations with simple examples, correctness and complexity analysis, computational and implementation considerations) and to many applications. I think it is overall the best introduction. <span class="citation" data-cites="vanderbei2014_linear">Vanderbei (<a href="#ref-vanderbei2014_linear">2014</a>)</span> follows a very similar outline, but contains more recent computational considerations<span><label for="sn-5" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-5" class="margin-toggle" /><span class="sidenote">For all the details about practical implementations of the simplex algorithm, <span class="citation" data-cites="maros2003_comput">Maros (<a href="#ref-maros2003_comput">2003</a>)</span> is dedicated to the computational aspects and contains everything you will need.<br />
|
||||||
|
<br />
|
||||||
|
</span></span>. (The author also has <a href="http://vanderbei.princeton.edu/307/lectures.html">lecture slides</a>.)</p>
|
||||||
|
<p>For more books on linear programming, the two books <span class="citation" data-cites="dantzig1997_linear">Dantzig (<a href="#ref-dantzig1997_linear">1997</a>)</span>, <span class="citation" data-cites="dantzig2003_linear">Dantzig (<a href="#ref-dantzig2003_linear">2003</a>)</span> are very complete, if somewhat more mathematically advanced. <span class="citation" data-cites="bertsimas1997_introd">Bertsimas and Tsitsiklis (<a href="#ref-bertsimas1997_introd">1997</a>)</span> is also a great reference, if you can find it.</p>
|
||||||
|
<p>For all the other subfields, <a href="https://or.stackexchange.com/a/870">this great StackExchange answer</a> contains a lot of useful references, including most of the above. Of particular note are <span class="citation" data-cites="peyreComputationalOptimalTransport2019">Peyré and Cuturi (<a href="#ref-peyreComputationalOptimalTransport2019">2019</a>)</span> for optimal transport, <span class="citation" data-cites="boyd2004_convex">Boyd (<a href="#ref-boyd2004_convex">2004</a>)</span> for convex optimization (<a href="https://web.stanford.edu/~boyd/cvxbook/">freely available online</a>), and <span class="citation" data-cites="nocedal2006_numer">Nocedal (<a href="#ref-nocedal2006_numer">2006</a>)</span> for numerical optimization. <span class="citation" data-cites="kochenderfer2019_algor">Kochenderfer (<a href="#ref-kochenderfer2019_algor">2019</a>)</span><span><label for="sn-6" class="margin-toggle">⊕</label><input type="checkbox" id="sn-6" class="margin-toggle" /><span class="marginnote"><br />
|
||||||
|
<br />
|
||||||
|
<img src="../images/or_references/kochenderfer.jpg" width="200" /><br />
|
||||||
|
<br />
|
||||||
|
</span></span> is not in the list (because it is very recent) but is also excellent, with examples in Julia covering nearly every kind of optimization algorithms.</p>
|
||||||
|
<h3 id="online-courses">Online courses</h3>
|
||||||
|
<p>If you would like to watch video lectures, there are a few good opportunities freely available online, in particular on <a href="https://ocw.mit.edu/index.htm">MIT OpenCourseWare</a>. The list of courses at MIT is available <a href="https://orc.mit.edu/academics/course-offerings">on their webpage</a>. I haven’t actually looked in details at the courses content<span><label for="sn-7" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-7" class="margin-toggle" /><span class="sidenote">I am more comfortable reading books than watching lecture videos online. Although I liked attending classes during my studies, I do not have the same feeling in front of a video. When I read, I can re-read three times the same sentence, pause to look up something, or skim a few paragraphs. I find that the inability to do that with a video diminishes greatly my ability to concentrate.<br />
|
||||||
|
<br />
|
||||||
|
</span></span>, so I cannot vouch for them directly, but MIT courses are generally of excellent quality. Most courses are also taught by Bertsimas and Bertsekas, who are very famous and wrote many excellent books.</p>
|
||||||
|
<p>Of particular notes are:</p>
|
||||||
|
<ul>
|
||||||
|
<li><a href="https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-251j-introduction-to-mathematical-programming-fall-2009/">Introduction to Mathematical Programming</a>,</li>
|
||||||
|
<li><a href="https://ocw.mit.edu/courses/sloan-school-of-management/15-084j-nonlinear-programming-spring-2004/">Nonlinear Optimization</a>,</li>
|
||||||
|
<li><a href="https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-253-convex-analysis-and-optimization-spring-2012/">Convex Analysis and Optimization</a>,</li>
|
||||||
|
<li><a href="https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-972-algebraic-techniques-and-semidefinite-optimization-spring-2006/">Algebraic Techniques and Semidefinite Optimization</a>,</li>
|
||||||
|
<li><a href="https://ocw.mit.edu/courses/sloan-school-of-management/15-083j-integer-programming-and-combinatorial-optimization-fall-2009/">Integer Programming and Combinatorial Optimization</a>.</li>
|
||||||
|
</ul>
|
||||||
|
<p>Another interesting course I found online is <a href="https://www.ams.jhu.edu/~wcook12/dl/index.html">Deep Learning in Discrete Optimization</a>, at Johns Hopkins<span><label for="sn-8" class="margin-toggle">⊕</label><input type="checkbox" id="sn-8" class="margin-toggle" /><span class="marginnote"> It is taught by William Cook, who is the author of <a href="https://press.princeton.edu/books/paperback/9780691163529/in-pursuit-of-the-traveling-salesman"><em>In Pursuit of the Traveling Salesman</em></a>, a nice introduction to the TSP problem in a readable form.<br />
|
||||||
|
<br />
|
||||||
|
</span></span>. It contains an interesting overview of deep learning and integer programming, with a focus on connections, and applications to recent research areas in ML (reinforcement learning, attention, etc.).</p>
|
||||||
|
<h2 id="solvers-and-computational-resources">Solvers and computational resources <span id="solvers"></span></h2>
|
||||||
|
<p>When you start reading about modelling and algorithms, I recommend you try solving a few problems yourself, either by hand for small instances, or using an existing solver. It will allow you to follow the examples in books, while also practising your modelling skills. You will also get an intuition of what is difficult to model and to solve.</p>
|
||||||
|
<p>There are many solvers available, both free and commercial, with various capabilities. I recommend you use the fantastic <a href="https://github.com/JuliaOpt/JuMP.jl">JuMP</a><span><label for="sn-9" class="margin-toggle">⊕</label><input type="checkbox" id="sn-9" class="margin-toggle" /><span class="marginnote"><br />
|
||||||
|
<br />
|
||||||
|
<img src="../images/or_references/jump.svg" width="250" /><br />
|
||||||
|
<br />
|
||||||
|
</span></span> library for Julia, which exposes a domain-specific language for modelling, along with interfaces to nearly all major solver packages. (Even if you don’t know Julia, this is a great and easy way to start!) If you’d rather use Python, you can use Google’s <a href="https://developers.google.com/optimization/introduction/python">OR-Tools</a> or <a href="https://github.com/coin-or/pulp">PuLP</a> for linear programming.</p>
|
||||||
|
<p>Regarding solvers, there is a <a href="http://www.juliaopt.org/JuMP.jl/stable/installation/#Getting-Solvers-1">list of solvers</a> on JuMP’s documentation, with their capabilities and their license. Free solvers include <a href="https://www.gnu.org/software/glpk/">GLPK</a> (linear programming), <a href="https://github.com/coin-or/Ipopt">Ipopt</a> (non-linear programming), and <a href="https://scip.zib.de/">SCIP</a> (mixed-integer linear programming).</p>
|
||||||
|
<p>Commercial solvers often have better performance, and some of them propose a free academic license: <a href="https://www.mosek.com/">MOSEK</a>, <a href="https://www.gurobi.com/">Gurobi</a>, and <a href="https://www.ibm.com/analytics/cplex-optimizer">IBM CPLEX</a> in particular all offer free academic licenses and work very well with JuMP.</p>
|
||||||
|
<p>Another awesome resource is the <a href="https://neos-server.org/neos/">NEOS Server</a>. It offers free computing resources for numerical optimization, including all major free and commercial solvers! You can submit jobs on it in a standard format, or interface your favourite programming language with it. The fact that such an amazing resource exists for free, for everyone is extraordinary. They also have an accompanying book, the <a href="https://neos-guide.org/">NEOS Guide</a>, containing many case studies and description of problem types. The <a href="https://neos-guide.org/content/optimization-taxonomy">taxonomy</a> may be particularly useful.</p>
|
||||||
|
<h2 id="conclusion">Conclusion</h2>
|
||||||
|
<p>Operations research is a fascinating topic, and it has an abundant literature that makes it very easy to dive into the subject. If you are interested in algorithms, modelling for practical applications, or just wish to understand more, I hope to have given you the first steps to follow, start reading and experimenting.</p>
|
||||||
|
<h2 id="references" class="unnumbered">References</h2>
|
||||||
|
<div id="refs" class="references">
|
||||||
|
<div id="ref-bertsimas1997_introd">
|
||||||
|
<p>Bertsimas, Dimitris, and John N. Tsitsiklis. 1997. <em>Introduction to Linear Optimization</em>. Belmont, Massachusetts: Athena Scientific. <a href="http://www.athenasc.com/linoptbook.html" class="uri">http://www.athenasc.com/linoptbook.html</a>.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-boyd2004_convex">
|
||||||
|
<p>Boyd, Stephen. 2004. <em>Convex Optimization</em>. Cambridge, UK New York: Cambridge University Press.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-chvatal1983_linear">
|
||||||
|
<p>Chvátal, Vašek. 1983. <em>Linear Programming</em>. New York: W.H. Freeman.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-dantzig1997_linear">
|
||||||
|
<p>Dantzig, George. 1997. <em>Linear Programming 1: Introduction</em>. New York: Springer. <a href="https://www.springer.com/gp/book/9780387948331" class="uri">https://www.springer.com/gp/book/9780387948331</a>.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-dantzig2003_linear">
|
||||||
|
<p>———. 2003. <em>Linear Programming 2: Theory and Extensions</em>. New York: Springer. <a href="https://www.springer.com/gp/book/9780387986135" class="uri">https://www.springer.com/gp/book/9780387986135</a>.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-kochenderfer2019_algor">
|
||||||
|
<p>Kochenderfer, Mykel. 2019. <em>Algorithms for Optimization</em>. Cambridge, Massachusetts: The MIT Press.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-maros2003_comput">
|
||||||
|
<p>Maros, István. 2003. <em>Computational Techniques of the Simplex Method</em>. Boston: Kluwer Academic Publishers.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-nocedal2006_numer">
|
||||||
|
<p>Nocedal, Jorge. 2006. <em>Numerical Optimization</em>. New York: Springer. <a href="https://www.springer.com/gp/book/9780387303031" class="uri">https://www.springer.com/gp/book/9780387303031</a>.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-peyreComputationalOptimalTransport2019">
|
||||||
|
<p>Peyré, Gabriel, and Marco Cuturi. 2019. “Computational Optimal Transport.” <em>Foundations and Trends in Machine Learning</em> 11 (5-6): 355–206. <a href="https://doi.org/10.1561/2200000073" class="uri">https://doi.org/10.1561/2200000073</a>.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-vanderbei2014_linear">
|
||||||
|
<p>Vanderbei, Robert. 2014. <em>Linear Programming : Foundations and Extensions</em>. New York: Springer.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-wentzel1988_operat">
|
||||||
|
<p>Wentzel, Elena S. 1988. <em>Operations Research: A Methodological Approach</em>. Moscow: Mir publishers.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-williams2013_model">
|
||||||
|
<p>Williams, H. Paul. 2013. <em>Model Building in Mathematical Programming</em>. Chichester, West Sussex: Wiley. <a href="https://www.wiley.com/en-fr/Model+Building+in+Mathematical+Programming,+5th+Edition-p-9781118443330" class="uri">https://www.wiley.com/en-fr/Model+Building+in+Mathematical+Programming,+5th+Edition-p-9781118443330</a>.</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
</article>
|
||||||
|
|
||||||
|
|
||||||
|
<footer>
|
||||||
|
Site proudly generated by
|
||||||
|
<a href="http://jaspervdj.be/hakyll">Hakyll</a>
|
||||||
|
</footer>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
117
_site/rss.xml
117
_site/rss.xml
|
|
@ -7,8 +7,123 @@
|
||||||
<description><![CDATA[Recent posts]]></description>
|
<description><![CDATA[Recent posts]]></description>
|
||||||
<atom:link href="https://www.lozeve.com/rss.xml" rel="self"
|
<atom:link href="https://www.lozeve.com/rss.xml" rel="self"
|
||||||
type="application/rss+xml" />
|
type="application/rss+xml" />
|
||||||
<lastBuildDate>Tue, 05 May 2020 00:00:00 UT</lastBuildDate>
|
<lastBuildDate>Wed, 27 May 2020 00:00:00 UT</lastBuildDate>
|
||||||
<item>
|
<item>
|
||||||
|
<title>Operations Research and Optimization: where to start?</title>
|
||||||
|
<link>https://www.lozeve.com/posts/operations-research-references.html</link>
|
||||||
|
<description><![CDATA[<article>
|
||||||
|
<section class="header">
|
||||||
|
|
||||||
|
</section>
|
||||||
|
<section>
|
||||||
|
<p><a href="https://en.wikipedia.org/wiki/Operations_research">Operations research</a> (OR) is a vast area comprising a lot of theory, different branches of mathematics, and too many applications to count. In this post, I will try to explain why it can be a little disconcerting to explore at first, and how to start investigating the topic with a few references to get started.</p>
|
||||||
|
<p>Keep in mind that although I studied it during my graduate studies, this is not my primary area of expertise (I’m a data scientist by trade), and I definitely don’t pretend to know everything in OR. This is a field too vast for any single person to understand in its entirety, and I talk mostly from an “amateur mathematician and computer scientist” standpoint.</p>
|
||||||
|
<h2 id="why-is-it-hard-to-approach">Why is it hard to approach?</h2>
|
||||||
|
<p>Operations research can be difficult to approach, since there are many references and subfields. Compared to machine learning for instance, OR has a slightly longer history (going back to the 17th century, for example with <a href="https://en.wikipedia.org/wiki/Gaspard_Monge">Monge</a> and the <a href="https://en.wikipedia.org/wiki/Transportation_theory_(mathematics)">optimal transport problem</a>)<span><label for="sn-1" class="margin-toggle">⊕</label><input type="checkbox" id="sn-1" class="margin-toggle" /><span class="marginnote"> For a very nice introduction (in French) to optimal transport, see these blog posts by <a href="https://twitter.com/gabrielpeyre">Gabriel Peyré</a>, on the CNRS maths blog: <a href="https://images.math.cnrs.fr/Le-transport-optimal-numerique-et-ses-applications-Partie-1.html">Part 1</a> and <a href="https://images.math.cnrs.fr/Le-transport-optimal-numerique-et-ses-applications-Partie-2.html">Part 2</a>. See also the resources on <a href="https://optimaltransport.github.io/">optimaltransport.github.io</a> (in English).<br />
|
||||||
|
<br />
|
||||||
|
</span></span>. This means that good textbooks and such have existed for a long time, but also that there will be plenty of material to choose from.</p>
|
||||||
|
<p>Moreover, OR is very close to applications. Sometimes methods may vary a lot in their presentation depending on whether they’re applied to train tracks, sudoku, or travelling salesmen. In practice, the terminology and notations are not the same everywhere. This is disconcerting if you are used to “pure” mathematics, where notations evolved over a long time and is pretty much standardised for many areas. In contrast, if you’re used to the statistics literature with its <a href="https://lingpipe-blog.com/2009/10/13/whats-wrong-with-probability-notation/">strange notations</a>, you will find that OR is actually very well formalized.</p>
|
||||||
|
<p>There are many subfields of operations research, including all kinds of optimization (constrained and unconstrained), game theory, dynamic programming, stochastic processes, etc.</p>
|
||||||
|
<h2 id="where-to-start">Where to start</h2>
|
||||||
|
<h3 id="introduction-and-modelling">Introduction and modelling</h3>
|
||||||
|
<p>For an overall introduction, I recommend <span class="citation" data-cites="wentzel1988_operat">Wentzel (<a href="#ref-wentzel1988_operat">1988</a>)</span>. It is an old book, published by Mir Publications, a Soviet publisher which published many excellent scientific textbooks<span><label for="sn-2" class="margin-toggle">⊕</label><input type="checkbox" id="sn-2" class="margin-toggle" /><span class="marginnote"> Mir also published <a href="https://mirtitles.org/2011/06/03/physics-for-everyone/"><em>Physics for Everyone</em></a> by Lev Landau and Alexander Kitaigorodsky, a three-volume introduction to physics that is really accessible. Together with Feynman’s famous <a href="https://www.feynmanlectures.caltech.edu/">lectures</a>, I read them (in French) when I was a kid, and it was the best introduction I could possibly have to the subject.<br />
|
||||||
|
<br />
|
||||||
|
</span></span>. It is out of print, but it is available <a href="https://archive.org/details/WentzelOperationsResearchMir1983">on Archive.org</a><span><label for="sn-3" class="margin-toggle">⊕</label><input type="checkbox" id="sn-3" class="margin-toggle" /><span class="marginnote"><br />
|
||||||
|
<br />
|
||||||
|
<img src="/images/or_references/wentzel.jpg" width="200" /><br />
|
||||||
|
<br />
|
||||||
|
</span></span>. The book is quite old, but everything presented is still extremely relevant today. It requires absolutely no background, and covers everything: a general introduction to the field, linear programming, dynamic programming, Markov processes and queues, Monte Carlo methods, and game theory. Even if you already know some of these topics, the presentations is so clear that it is a pleasure to read! (In particular, it is one of the best presentations of dynamic programming that I have ever read. The explanation of the simplex algorithm is also excellent.)</p>
|
||||||
|
<p>If you are interested in optimization, the first thing you have to learn is modelling, i.e. transforming your problem (described in natural language, often from a particular industrial application) into a mathematical programme. The mathematical programme is the structure on which you will be able to apply an algorithm to find an optimal solution. Even if (like me) you are initially more interested in the algorithmic side of things, learning to create models will shed a lot of light on the overall process, and will give you more insight in general on the reasoning behind algorithms.</p>
|
||||||
|
<p>The best book I have read on the subject is <span class="citation" data-cites="williams2013_model">Williams (<a href="#ref-williams2013_model">2013</a>)</span><span><label for="sn-4" class="margin-toggle">⊕</label><input type="checkbox" id="sn-4" class="margin-toggle" /><span class="marginnote"><br />
|
||||||
|
<br />
|
||||||
|
<img src="/images/or_references/williams.jpg" width="200" /><br />
|
||||||
|
<br />
|
||||||
|
</span></span>. It contains a lot of concrete, step-by-step examples on concrete applications, in a multitude of domains, and remains very easy to read and to follow. It covers nearly every type of problem, so it is very useful as a reference. When you encounter a concrete problem in real life afterwards, you will know how to construct an appropriate model, and in the process you will often identify a common type of problem. The book then gives plenty of advice on how to approach each type of problem. Finally, it is also a great resource to build a “mental map” of the field, avoiding getting lost in the jungle of linear, stochastic, mixed integer, quadratic, and other network problems.</p>
|
||||||
|
<p>Another interesting resource is the freely available <a href="https://docs.mosek.com/modeling-cookbook/index.html">MOSEK Modeling Cookbook</a>, covering many types of problems, with more mathematical details than in <span class="citation" data-cites="williams2013_model">Williams (<a href="#ref-williams2013_model">2013</a>)</span>. It is built for people wanting to use the commercial <a href="https://www.mosek.com/">MOSEK</a> solver, so it could be useful if you plan to use a solver package like this one (more details on solvers <a href="#solvers">below</a>).</p>
|
||||||
|
<h3 id="theory-and-algorithms">Theory and algorithms</h3>
|
||||||
|
<p>The basic algorithm for optimization is the <a href="https://en.wikipedia.org/wiki/Simplex_algorithm">simplex algorithm</a>, developed by Dantzig in the 1940s to solve <a href="https://en.wikipedia.org/wiki/Linear_programming">linear programming</a> problems. It is the one of the main building blocks for mathematical optimization, and is used and referenced extensively in all kinds of approaches. As such, it is really important to understand it in detail. There are many books on the subject, but I especially liked <span class="citation" data-cites="chvatal1983_linear">Chvátal (<a href="#ref-chvatal1983_linear">1983</a>)</span> (out of print, but you can find cheap used versions on Amazon). It covers everything there is to know on the simplex algorithms (step-by-step explanations with simple examples, correctness and complexity analysis, computational and implementation considerations) and to many applications. I think it is overall the best introduction. <span class="citation" data-cites="vanderbei2014_linear">Vanderbei (<a href="#ref-vanderbei2014_linear">2014</a>)</span> follows a very similar outline, but contains more recent computational considerations<span><label for="sn-5" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-5" class="margin-toggle" /><span class="sidenote">For all the details about practical implementations of the simplex algorithm, <span class="citation" data-cites="maros2003_comput">Maros (<a href="#ref-maros2003_comput">2003</a>)</span> is dedicated to the computational aspects and contains everything you will need.<br />
|
||||||
|
<br />
|
||||||
|
</span></span>. (The author also has <a href="http://vanderbei.princeton.edu/307/lectures.html">lecture slides</a>.)</p>
|
||||||
|
<p>For more books on linear programming, the two books <span class="citation" data-cites="dantzig1997_linear">Dantzig (<a href="#ref-dantzig1997_linear">1997</a>)</span>, <span class="citation" data-cites="dantzig2003_linear">Dantzig (<a href="#ref-dantzig2003_linear">2003</a>)</span> are very complete, if somewhat more mathematically advanced. <span class="citation" data-cites="bertsimas1997_introd">Bertsimas and Tsitsiklis (<a href="#ref-bertsimas1997_introd">1997</a>)</span> is also a great reference, if you can find it.</p>
|
||||||
|
<p>For all the other subfields, <a href="https://or.stackexchange.com/a/870">this great StackExchange answer</a> contains a lot of useful references, including most of the above. Of particular note are <span class="citation" data-cites="peyreComputationalOptimalTransport2019">Peyré and Cuturi (<a href="#ref-peyreComputationalOptimalTransport2019">2019</a>)</span> for optimal transport, <span class="citation" data-cites="boyd2004_convex">Boyd (<a href="#ref-boyd2004_convex">2004</a>)</span> for convex optimization (<a href="https://web.stanford.edu/~boyd/cvxbook/">freely available online</a>), and <span class="citation" data-cites="nocedal2006_numer">Nocedal (<a href="#ref-nocedal2006_numer">2006</a>)</span> for numerical optimization. <span class="citation" data-cites="kochenderfer2019_algor">Kochenderfer (<a href="#ref-kochenderfer2019_algor">2019</a>)</span><span><label for="sn-6" class="margin-toggle">⊕</label><input type="checkbox" id="sn-6" class="margin-toggle" /><span class="marginnote"><br />
|
||||||
|
<br />
|
||||||
|
<img src="/images/or_references/kochenderfer.jpg" width="200" /><br />
|
||||||
|
<br />
|
||||||
|
</span></span> is not in the list (because it is very recent) but is also excellent, with examples in Julia covering nearly every kind of optimization algorithms.</p>
|
||||||
|
<h3 id="online-courses">Online courses</h3>
|
||||||
|
<p>If you would like to watch video lectures, there are a few good opportunities freely available online, in particular on <a href="https://ocw.mit.edu/index.htm">MIT OpenCourseWare</a>. The list of courses at MIT is available <a href="https://orc.mit.edu/academics/course-offerings">on their webpage</a>. I haven’t actually looked in details at the courses content<span><label for="sn-7" class="margin-toggle sidenote-number"></label><input type="checkbox" id="sn-7" class="margin-toggle" /><span class="sidenote">I am more comfortable reading books than watching lecture videos online. Although I liked attending classes during my studies, I do not have the same feeling in front of a video. When I read, I can re-read three times the same sentence, pause to look up something, or skim a few paragraphs. I find that the inability to do that with a video diminishes greatly my ability to concentrate.<br />
|
||||||
|
<br />
|
||||||
|
</span></span>, so I cannot vouch for them directly, but MIT courses are generally of excellent quality. Most courses are also taught by Bertsimas and Bertsekas, who are very famous and wrote many excellent books.</p>
|
||||||
|
<p>Of particular notes are:</p>
|
||||||
|
<ul>
|
||||||
|
<li><a href="https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-251j-introduction-to-mathematical-programming-fall-2009/">Introduction to Mathematical Programming</a>,</li>
|
||||||
|
<li><a href="https://ocw.mit.edu/courses/sloan-school-of-management/15-084j-nonlinear-programming-spring-2004/">Nonlinear Optimization</a>,</li>
|
||||||
|
<li><a href="https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-253-convex-analysis-and-optimization-spring-2012/">Convex Analysis and Optimization</a>,</li>
|
||||||
|
<li><a href="https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-972-algebraic-techniques-and-semidefinite-optimization-spring-2006/">Algebraic Techniques and Semidefinite Optimization</a>,</li>
|
||||||
|
<li><a href="https://ocw.mit.edu/courses/sloan-school-of-management/15-083j-integer-programming-and-combinatorial-optimization-fall-2009/">Integer Programming and Combinatorial Optimization</a>.</li>
|
||||||
|
</ul>
|
||||||
|
<p>Another interesting course I found online is <a href="https://www.ams.jhu.edu/~wcook12/dl/index.html">Deep Learning in Discrete Optimization</a>, at Johns Hopkins<span><label for="sn-8" class="margin-toggle">⊕</label><input type="checkbox" id="sn-8" class="margin-toggle" /><span class="marginnote"> It is taught by William Cook, who is the author of <a href="https://press.princeton.edu/books/paperback/9780691163529/in-pursuit-of-the-traveling-salesman"><em>In Pursuit of the Traveling Salesman</em></a>, a nice introduction to the TSP problem in a readable form.<br />
|
||||||
|
<br />
|
||||||
|
</span></span>. It contains an interesting overview of deep learning and integer programming, with a focus on connections, and applications to recent research areas in ML (reinforcement learning, attention, etc.).</p>
|
||||||
|
<h2 id="solvers-and-computational-resources">Solvers and computational resources <span id="solvers"></span></h2>
|
||||||
|
<p>When you start reading about modelling and algorithms, I recommend you try solving a few problems yourself, either by hand for small instances, or using an existing solver. It will allow you to follow the examples in books, while also practising your modelling skills. You will also get an intuition of what is difficult to model and to solve.</p>
|
||||||
|
<p>There are many solvers available, both free and commercial, with various capabilities. I recommend you use the fantastic <a href="https://github.com/JuliaOpt/JuMP.jl">JuMP</a><span><label for="sn-9" class="margin-toggle">⊕</label><input type="checkbox" id="sn-9" class="margin-toggle" /><span class="marginnote"><br />
|
||||||
|
<br />
|
||||||
|
<img src="/images/or_references/jump.svg" width="250" /><br />
|
||||||
|
<br />
|
||||||
|
</span></span> library for Julia, which exposes a domain-specific language for modelling, along with interfaces to nearly all major solver packages. (Even if you don’t know Julia, this is a great and easy way to start!) If you’d rather use Python, you can use Google’s <a href="https://developers.google.com/optimization/introduction/python">OR-Tools</a> or <a href="https://github.com/coin-or/pulp">PuLP</a> for linear programming.</p>
|
||||||
|
<p>Regarding solvers, there is a <a href="http://www.juliaopt.org/JuMP.jl/stable/installation/#Getting-Solvers-1">list of solvers</a> on JuMP’s documentation, with their capabilities and their license. Free solvers include <a href="https://www.gnu.org/software/glpk/">GLPK</a> (linear programming), <a href="https://github.com/coin-or/Ipopt">Ipopt</a> (non-linear programming), and <a href="https://scip.zib.de/">SCIP</a> (mixed-integer linear programming).</p>
|
||||||
|
<p>Commercial solvers often have better performance, and some of them propose a free academic license: <a href="https://www.mosek.com/">MOSEK</a>, <a href="https://www.gurobi.com/">Gurobi</a>, and <a href="https://www.ibm.com/analytics/cplex-optimizer">IBM CPLEX</a> in particular all offer free academic licenses and work very well with JuMP.</p>
|
||||||
|
<p>Another awesome resource is the <a href="https://neos-server.org/neos/">NEOS Server</a>. It offers free computing resources for numerical optimization, including all major free and commercial solvers! You can submit jobs on it in a standard format, or interface your favourite programming language with it. The fact that such an amazing resource exists for free, for everyone is extraordinary. They also have an accompanying book, the <a href="https://neos-guide.org/">NEOS Guide</a>, containing many case studies and description of problem types. The <a href="https://neos-guide.org/content/optimization-taxonomy">taxonomy</a> may be particularly useful.</p>
|
||||||
|
<h2 id="conclusion">Conclusion</h2>
|
||||||
|
<p>Operations research is a fascinating topic, and it has an abundant literature that makes it very easy to dive into the subject. If you are interested in algorithms, modelling for practical applications, or just wish to understand more, I hope to have given you the first steps to follow, start reading and experimenting.</p>
|
||||||
|
<h2 id="references" class="unnumbered">References</h2>
|
||||||
|
<div id="refs" class="references">
|
||||||
|
<div id="ref-bertsimas1997_introd">
|
||||||
|
<p>Bertsimas, Dimitris, and John N. Tsitsiklis. 1997. <em>Introduction to Linear Optimization</em>. Belmont, Massachusetts: Athena Scientific. <a href="http://www.athenasc.com/linoptbook.html" class="uri">http://www.athenasc.com/linoptbook.html</a>.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-boyd2004_convex">
|
||||||
|
<p>Boyd, Stephen. 2004. <em>Convex Optimization</em>. Cambridge, UK New York: Cambridge University Press.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-chvatal1983_linear">
|
||||||
|
<p>Chvátal, Vašek. 1983. <em>Linear Programming</em>. New York: W.H. Freeman.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-dantzig1997_linear">
|
||||||
|
<p>Dantzig, George. 1997. <em>Linear Programming 1: Introduction</em>. New York: Springer. <a href="https://www.springer.com/gp/book/9780387948331" class="uri">https://www.springer.com/gp/book/9780387948331</a>.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-dantzig2003_linear">
|
||||||
|
<p>———. 2003. <em>Linear Programming 2: Theory and Extensions</em>. New York: Springer. <a href="https://www.springer.com/gp/book/9780387986135" class="uri">https://www.springer.com/gp/book/9780387986135</a>.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-kochenderfer2019_algor">
|
||||||
|
<p>Kochenderfer, Mykel. 2019. <em>Algorithms for Optimization</em>. Cambridge, Massachusetts: The MIT Press.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-maros2003_comput">
|
||||||
|
<p>Maros, István. 2003. <em>Computational Techniques of the Simplex Method</em>. Boston: Kluwer Academic Publishers.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-nocedal2006_numer">
|
||||||
|
<p>Nocedal, Jorge. 2006. <em>Numerical Optimization</em>. New York: Springer. <a href="https://www.springer.com/gp/book/9780387303031" class="uri">https://www.springer.com/gp/book/9780387303031</a>.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-peyreComputationalOptimalTransport2019">
|
||||||
|
<p>Peyré, Gabriel, and Marco Cuturi. 2019. “Computational Optimal Transport.” <em>Foundations and Trends in Machine Learning</em> 11 (5-6): 355–206. <a href="https://doi.org/10.1561/2200000073" class="uri">https://doi.org/10.1561/2200000073</a>.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-vanderbei2014_linear">
|
||||||
|
<p>Vanderbei, Robert. 2014. <em>Linear Programming : Foundations and Extensions</em>. New York: Springer.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-wentzel1988_operat">
|
||||||
|
<p>Wentzel, Elena S. 1988. <em>Operations Research: A Methodological Approach</em>. Moscow: Mir publishers.</p>
|
||||||
|
</div>
|
||||||
|
<div id="ref-williams2013_model">
|
||||||
|
<p>Williams, H. Paul. 2013. <em>Model Building in Mathematical Programming</em>. Chichester, West Sussex: Wiley. <a href="https://www.wiley.com/en-fr/Model+Building+in+Mathematical+Programming,+5th+Edition-p-9781118443330" class="uri">https://www.wiley.com/en-fr/Model+Building+in+Mathematical+Programming,+5th+Edition-p-9781118443330</a>.</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
</article>
|
||||||
|
]]></description>
|
||||||
|
<pubDate>Wed, 27 May 2020 00:00:00 UT</pubDate>
|
||||||
|
<guid>https://www.lozeve.com/posts/operations-research-references.html</guid>
|
||||||
|
<dc:creator>Dimitri Lozeve</dc:creator>
|
||||||
|
</item>
|
||||||
|
<item>
|
||||||
<title>ICLR 2020 Notes: Speakers and Workshops</title>
|
<title>ICLR 2020 Notes: Speakers and Workshops</title>
|
||||||
<link>https://www.lozeve.com/posts/iclr-2020-notes.html</link>
|
<link>https://www.lozeve.com/posts/iclr-2020-notes.html</link>
|
||||||
<description><
|
|
@ -0,0 +1 @@
|
||||||
|
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 377.42 122.03"><defs><style>.cls-1{fill:#231f20;}.cls-2,.cls-3,.cls-5,.cls-7,.cls-9{fill:none;}.cls-2,.cls-3{stroke:#231f20;stroke-linecap:round;stroke-linejoin:round;}.cls-2{stroke-width:6.63px;}.cls-3{stroke-width:6.63px;fill-rule:evenodd;}.cls-4{fill:#aa7dc0;}.cls-5{stroke:#945bb0;}.cls-5,.cls-7,.cls-9{stroke-miterlimit:10;stroke-width:3px;}.cls-6{fill:#6bab5b;}.cls-7{stroke:#3b972e;}.cls-8{fill:#d66661;}.cls-9{stroke:#c93d39;}</style></defs><title>Thin-Border-Logo-Text</title><g id="Page_1" data-name="Page 1"><g id="Layer_1-2" data-name="Layer 1"><polygon class="cls-1" points="313.71 117.21 326.3 117.21 326.3 50.53 311.9 50.53 291.01 79.2 270.12 50.53 255.62 50.53 255.62 117.21 268.21 117.21 268.21 68.72 289.48 97.78 291.96 97.78 313.71 68.81 313.71 117.21 313.71 117.21"/><path class="cls-1" d="M172.45,50.54H185.2V93.37c0,3.86.29,7.71-.65,11.5a19.26,19.26,0,0,1-7.72,11.33,19.75,19.75,0,0,1-9.07,3.42,20.85,20.85,0,0,1-22.67-16.11l12.38-3.16a8.45,8.45,0,0,0,5.17,6.09,8,8,0,0,0,3.67.45,7.09,7.09,0,0,0,3.2-1.16c2.77-1.93,2.94-5.34,2.94-8.34V50.54Zm26.46,0h12.52V74.35c0,4.61.31,7.84.95,9.63a8.46,8.46,0,0,0,3.06,4.19,8.76,8.76,0,0,0,5.2,1.51,9.15,9.15,0,0,0,5.24-1.47,8.68,8.68,0,0,0,3.2-4.34q.76-2.13.76-9.12V50.57h12.43V71.49c0,8.61-.69,14.52-2,17.69a19.21,19.21,0,0,1-7.34,8.9q-4.86,3.11-12.34,3.11-8.13,0-13.13-3.62a19.27,19.27,0,0,1-7.06-10.12c-.94-3-1.45-8.44-1.45-16.32V50.57Zm138.87,0h13.46q11,0,15.76,2a16,16,0,0,1,7.63,6.54,20.21,20.21,0,0,1,2.79,10.8,18.23,18.23,0,0,1-3.65,11.61A18.82,18.82,0,0,1,363.86,88q-3.66,1-13.37,1v28.26H337.78V50.54Zm12.71,26.08h4a22.42,22.42,0,0,0,6.63-.68,5.82,5.82,0,0,0,2.92-2.25,6.53,6.53,0,0,0,1.07-3.79,6.09,6.09,0,0,0-3-5.62C360.69,63.41,358,63,354.06,63h-3.57V76.62Z"/><line class="cls-2" x1="19.64" y1="67.28" x2="120.16" y2="28.42"/><polyline class="cls-3" points="3.32 118.72 89.75 32.29 118.72 3.32"/><path class="cls-4" d="M106.29,120.18a17.17,17.17,0,1,1,12.16-5A17.06,17.06,0,0,1,106.29,120.18Z"/><path class="cls-5" d="M106.29,120.18a17.17,17.17,0,1,1,12.16-5A17.06,17.06,0,0,1,106.29,120.18Z"/><path class="cls-6" d="M106.29,79.33a17.19,17.19,0,1,1,12.16-5A17.07,17.07,0,0,1,106.29,79.33Z"/><path class="cls-7" d="M106.29,79.33a17.19,17.19,0,1,1,12.16-5A17.07,17.07,0,0,1,106.29,79.33Z"/><path class="cls-8" d="M65.28,120.18a17.19,17.19,0,1,1,12.15-5A17,17,0,0,1,65.28,120.18Z"/><path class="cls-9" d="M65.28,120.18a17.19,17.19,0,1,1,12.15-5A17,17,0,0,1,65.28,120.18Z"/></g></g></svg>
|
||||||
|
After Width: | Height: | Size: 2.5 KiB |
BIN
images/or_references/kochenderfer.jpg
Normal file
BIN
images/or_references/kochenderfer.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 54 KiB |
BIN
images/or_references/wentzel.jpg
Normal file
BIN
images/or_references/wentzel.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 108 KiB |
BIN
images/or_references/williams.jpg
Normal file
BIN
images/or_references/williams.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 52 KiB |
243
posts/operations-research-references.org
Normal file
243
posts/operations-research-references.org
Normal file
|
|
@ -0,0 +1,243 @@
|
||||||
|
---
|
||||||
|
title: "Operations Research and Optimization: where to start?"
|
||||||
|
date: 2020-05-27
|
||||||
|
---
|
||||||
|
|
||||||
|
[[https://en.wikipedia.org/wiki/Operations_research][Operations research]] (OR) is a vast area comprising a lot of theory,
|
||||||
|
different branches of mathematics, and too many applications to
|
||||||
|
count. In this post, I will try to explain why it can be a little
|
||||||
|
disconcerting to explore at first, and how to start investigating the
|
||||||
|
topic with a few references to get started.
|
||||||
|
|
||||||
|
Keep in mind that although I studied it during my graduate studies,
|
||||||
|
this is not my primary area of expertise (I'm a data scientist by
|
||||||
|
trade), and I definitely don't pretend to know everything in OR. This
|
||||||
|
is a field too vast for any single person to understand in its
|
||||||
|
entirety, and I talk mostly from an "amateur mathematician and
|
||||||
|
computer scientist" standpoint.
|
||||||
|
|
||||||
|
* Why is it hard to approach?
|
||||||
|
|
||||||
|
Operations research can be difficult to approach, since there are many
|
||||||
|
references and subfields. Compared to machine learning for instance,
|
||||||
|
OR has a slightly longer history (going back to the 17th century, for
|
||||||
|
example with [[https://en.wikipedia.org/wiki/Gaspard_Monge][Monge]] and the [[https://en.wikipedia.org/wiki/Transportation_theory_(mathematics)][optimal transport
|
||||||
|
problem]])[fn:optimaltransport]. This means that good textbooks and such
|
||||||
|
have existed for a long time, but also that there will be plenty of
|
||||||
|
material to choose from.
|
||||||
|
|
||||||
|
[fn:optimaltransport] {-} For a very nice introduction (in French) to
|
||||||
|
optimal transport, see these blog posts by [[https://twitter.com/gabrielpeyre][Gabriel Peyré]], on the CNRS
|
||||||
|
maths blog: [[https://images.math.cnrs.fr/Le-transport-optimal-numerique-et-ses-applications-Partie-1.html][Part 1]] and [[https://images.math.cnrs.fr/Le-transport-optimal-numerique-et-ses-applications-Partie-2.html][Part 2]]. See also the resources on
|
||||||
|
[[https://optimaltransport.github.io/][optimaltransport.github.io]] (in English).
|
||||||
|
|
||||||
|
|
||||||
|
Moreover, OR is very close to applications. Sometimes methods may vary
|
||||||
|
a lot in their presentation depending on whether they're applied to
|
||||||
|
train tracks, sudoku, or travelling salesmen. In practice, the
|
||||||
|
terminology and notations are not the same everywhere. This is
|
||||||
|
disconcerting if you are used to "pure" mathematics, where notations
|
||||||
|
evolved over a long time and is pretty much standardised for many
|
||||||
|
areas. In contrast, if you're used to the statistics literature with
|
||||||
|
its [[https://lingpipe-blog.com/2009/10/13/whats-wrong-with-probability-notation/][strange notations]], you will find that OR is actually very well
|
||||||
|
formalized.
|
||||||
|
|
||||||
|
There are many subfields of operations research, including all kinds
|
||||||
|
of optimization (constrained and unconstrained), game theory, dynamic
|
||||||
|
programming, stochastic processes, etc.
|
||||||
|
|
||||||
|
* Where to start
|
||||||
|
|
||||||
|
** Introduction and modelling
|
||||||
|
|
||||||
|
For an overall introduction, I recommend cite:wentzel1988_operat. It
|
||||||
|
is an old book, published by Mir Publications, a Soviet publisher
|
||||||
|
which published many excellent scientific textbooks[fn:mir]. It is out
|
||||||
|
of print, but it is available [[https://archive.org/details/WentzelOperationsResearchMir1983][on Archive.org]][fn:wentzel]. The book is
|
||||||
|
quite old, but everything presented is still extremely relevant
|
||||||
|
today. It requires absolutely no background, and covers everything: a
|
||||||
|
general introduction to the field, linear programming, dynamic
|
||||||
|
programming, Markov processes and queues, Monte Carlo methods, and
|
||||||
|
game theory. Even if you already know some of these topics, the
|
||||||
|
presentations is so clear that it is a pleasure to read! (In
|
||||||
|
particular, it is one of the best presentations of dynamic programming
|
||||||
|
that I have ever read. The explanation of the simplex algorithm is
|
||||||
|
also excellent.)
|
||||||
|
|
||||||
|
[fn:wentzel] {-}
|
||||||
|
#+ATTR_HTML: :width 200px
|
||||||
|
[[file:/images/or_references/wentzel.jpg]]
|
||||||
|
|
||||||
|
[fn:mir] {-} Mir also published [[https://mirtitles.org/2011/06/03/physics-for-everyone/][/Physics for Everyone/]] by Lev Landau
|
||||||
|
and Alexander Kitaigorodsky, a three-volume introduction to physics
|
||||||
|
that is really accessible. Together with Feynman's famous [[https://www.feynmanlectures.caltech.edu/][lectures]], I
|
||||||
|
read them (in French) when I was a kid, and it was the best
|
||||||
|
introduction I could possibly have to the subject.
|
||||||
|
|
||||||
|
|
||||||
|
If you are interested in optimization, the first thing you have to
|
||||||
|
learn is modelling, i.e. transforming your problem (described in
|
||||||
|
natural language, often from a particular industrial application) into
|
||||||
|
a mathematical programme. The mathematical programme is the structure
|
||||||
|
on which you will be able to apply an algorithm to find an optimal
|
||||||
|
solution. Even if (like me) you are initially more interested in the
|
||||||
|
algorithmic side of things, learning to create models will shed a lot
|
||||||
|
of light on the overall process, and will give you more insight in
|
||||||
|
general on the reasoning behind algorithms.
|
||||||
|
|
||||||
|
The best book I have read on the subject is
|
||||||
|
cite:williams2013_model[fn:williams]. It contains a lot of concrete,
|
||||||
|
step-by-step examples on concrete applications, in a multitude of
|
||||||
|
domains, and remains very easy to read and to follow. It covers nearly
|
||||||
|
every type of problem, so it is very useful as a reference. When you
|
||||||
|
encounter a concrete problem in real life afterwards, you will know
|
||||||
|
how to construct an appropriate model, and in the process you will
|
||||||
|
often identify a common type of problem. The book then gives plenty of
|
||||||
|
advice on how to approach each type of problem. Finally, it is also a
|
||||||
|
great resource to build a "mental map" of the field, avoiding getting
|
||||||
|
lost in the jungle of linear, stochastic, mixed integer, quadratic,
|
||||||
|
and other network problems.
|
||||||
|
|
||||||
|
[fn:williams] {-}
|
||||||
|
#+ATTR_HTML: :width 200px
|
||||||
|
[[file:/images/or_references/williams.jpg]]
|
||||||
|
|
||||||
|
|
||||||
|
Another interesting resource is the freely available [[https://docs.mosek.com/modeling-cookbook/index.html][MOSEK Modeling
|
||||||
|
Cookbook]], covering many types of problems, with more mathematical
|
||||||
|
details than in cite:williams2013_model. It is built for people
|
||||||
|
wanting to use the commercial [[https://www.mosek.com/][MOSEK]] solver, so it could be useful if
|
||||||
|
you plan to use a solver package like this one (more details on
|
||||||
|
solvers [[solvers][below]]).
|
||||||
|
|
||||||
|
** Theory and algorithms
|
||||||
|
|
||||||
|
The basic algorithm for optimization is the [[https://en.wikipedia.org/wiki/Simplex_algorithm][simplex algorithm]],
|
||||||
|
developed by Dantzig in the 1940s to solve [[https://en.wikipedia.org/wiki/Linear_programming][linear programming]]
|
||||||
|
problems. It is the one of the main building blocks for mathematical
|
||||||
|
optimization, and is used and referenced extensively in all kinds of
|
||||||
|
approaches. As such, it is really important to understand it in
|
||||||
|
detail. There are many books on the subject, but I especially liked
|
||||||
|
cite:chvatal1983_linear (out of print, but you can find cheap used
|
||||||
|
versions on Amazon). It covers everything there is to know on the
|
||||||
|
simplex algorithms (step-by-step explanations with simple examples,
|
||||||
|
correctness and complexity analysis, computational and implementation
|
||||||
|
considerations) and to many applications. I think it is overall the
|
||||||
|
best introduction. cite:vanderbei2014_linear follows a very similar
|
||||||
|
outline, but contains more recent computational
|
||||||
|
considerations[fn:simplex_implem]. (The author also has [[http://vanderbei.princeton.edu/307/lectures.html][lecture
|
||||||
|
slides]].)
|
||||||
|
|
||||||
|
[fn:simplex_implem] For all the details about practical
|
||||||
|
implementations of the simplex algorithm, cite:maros2003_comput is
|
||||||
|
dedicated to the computational aspects and contains everything you
|
||||||
|
will need.
|
||||||
|
|
||||||
|
|
||||||
|
For more books on linear programming, the two books
|
||||||
|
cite:dantzig1997_linear, cite:dantzig2003_linear are very complete, if
|
||||||
|
somewhat more mathematically advanced. cite:bertsimas1997_introd is
|
||||||
|
also a great reference, if you can find it.
|
||||||
|
|
||||||
|
For all the other subfields, [[https://or.stackexchange.com/a/870][this great StackExchange answer]] contains
|
||||||
|
a lot of useful references, including most of the above. Of particular
|
||||||
|
note are cite:peyreComputationalOptimalTransport2019 for optimal
|
||||||
|
transport, cite:boyd2004_convex for convex optimization ([[https://web.stanford.edu/~boyd/cvxbook/][freely
|
||||||
|
available online]]), and cite:nocedal2006_numer for numerical
|
||||||
|
optimization. cite:kochenderfer2019_algor[fn:kochenderfer] is not in
|
||||||
|
the list (because it is very recent) but is also excellent, with
|
||||||
|
examples in Julia covering nearly every kind of optimization
|
||||||
|
algorithms.
|
||||||
|
|
||||||
|
[fn:kochenderfer] {-}
|
||||||
|
#+ATTR_HTML: :width 200px
|
||||||
|
[[file:/images/or_references/kochenderfer.jpg]]
|
||||||
|
|
||||||
|
|
||||||
|
** Online courses
|
||||||
|
|
||||||
|
If you would like to watch video lectures, there are a few good
|
||||||
|
opportunities freely available online, in particular on [[https://ocw.mit.edu/index.htm][MIT
|
||||||
|
OpenCourseWare]]. The list of courses at MIT is available [[https://orc.mit.edu/academics/course-offerings][on their
|
||||||
|
webpage]]. I haven't actually looked in details at the courses
|
||||||
|
content[fn:courses], so I cannot vouch for them directly, but MIT
|
||||||
|
courses are generally of excellent quality. Most courses are also
|
||||||
|
taught by Bertsimas and Bertsekas, who are very famous and wrote many
|
||||||
|
excellent books.
|
||||||
|
|
||||||
|
[fn:courses] I am more comfortable reading books than watching lecture
|
||||||
|
videos online. Although I liked attending classes during my studies, I
|
||||||
|
do not have the same feeling in front of a video. When I read, I can
|
||||||
|
re-read three times the same sentence, pause to look up something, or
|
||||||
|
skim a few paragraphs. I find that the inability to do that with a
|
||||||
|
video diminishes greatly my ability to concentrate.
|
||||||
|
|
||||||
|
|
||||||
|
Of particular notes are:
|
||||||
|
- [[https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-251j-introduction-to-mathematical-programming-fall-2009/][Introduction to Mathematical Programming]],
|
||||||
|
- [[https://ocw.mit.edu/courses/sloan-school-of-management/15-084j-nonlinear-programming-spring-2004/][Nonlinear Optimization]],
|
||||||
|
- [[https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-253-convex-analysis-and-optimization-spring-2012/][Convex Analysis and Optimization]],
|
||||||
|
- [[https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-972-algebraic-techniques-and-semidefinite-optimization-spring-2006/][Algebraic Techniques and Semidefinite Optimization]],
|
||||||
|
- [[https://ocw.mit.edu/courses/sloan-school-of-management/15-083j-integer-programming-and-combinatorial-optimization-fall-2009/][Integer Programming and Combinatorial Optimization]].
|
||||||
|
|
||||||
|
Another interesting course I found online is [[https://www.ams.jhu.edu/~wcook12/dl/index.html][Deep Learning in Discrete
|
||||||
|
Optimization]], at Johns Hopkins[fn:cook]. It contains an interesting
|
||||||
|
overview of deep learning and integer programming, with a focus on
|
||||||
|
connections, and applications to recent research areas in ML
|
||||||
|
(reinforcement learning, attention, etc.).
|
||||||
|
|
||||||
|
[fn:cook] {-} It is taught by William Cook, who is the author of [[https://press.princeton.edu/books/paperback/9780691163529/in-pursuit-of-the-traveling-salesman][/In
|
||||||
|
Pursuit of the Traveling Salesman/]], a nice introduction to the TSP
|
||||||
|
problem in a readable form.
|
||||||
|
|
||||||
|
|
||||||
|
* Solvers and computational resources <<solvers>>
|
||||||
|
|
||||||
|
When you start reading about modelling and algorithms, I recommend you
|
||||||
|
try solving a few problems yourself, either by hand for small
|
||||||
|
instances, or using an existing solver. It will allow you to follow
|
||||||
|
the examples in books, while also practising your modelling
|
||||||
|
skills. You will also get an intuition of what is difficult to model
|
||||||
|
and to solve.
|
||||||
|
|
||||||
|
There are many solvers available, both free and commercial, with
|
||||||
|
various capabilities. I recommend you use the fantastic [[https://github.com/JuliaOpt/JuMP.jl][JuMP]][fn:jump]
|
||||||
|
library for Julia, which exposes a domain-specific language for
|
||||||
|
modelling, along with interfaces to nearly all major solver
|
||||||
|
packages. (Even if you don't know Julia, this is a great and easy way
|
||||||
|
to start!) If you'd rather use Python, you can use Google's [[https://developers.google.com/optimization/introduction/python][OR-Tools]]
|
||||||
|
or [[https://github.com/coin-or/pulp][PuLP]] for linear programming.
|
||||||
|
|
||||||
|
[fn:jump] {-}
|
||||||
|
#+ATTR_HTML: :width 250px
|
||||||
|
[[file:/images/or_references/jump.svg]]
|
||||||
|
|
||||||
|
|
||||||
|
Regarding solvers, there is a [[http://www.juliaopt.org/JuMP.jl/stable/installation/#Getting-Solvers-1][list of solvers]] on JuMP's documentation,
|
||||||
|
with their capabilities and their license. Free solvers include [[https://www.gnu.org/software/glpk/][GLPK]]
|
||||||
|
(linear programming), [[https://github.com/coin-or/Ipopt][Ipopt]] (non-linear programming), and [[https://scip.zib.de/][SCIP]]
|
||||||
|
(mixed-integer linear programming).
|
||||||
|
|
||||||
|
Commercial solvers often have better performance, and some of them
|
||||||
|
propose a free academic license: [[https://www.mosek.com/][MOSEK]], [[https://www.gurobi.com/][Gurobi]], and [[https://www.ibm.com/analytics/cplex-optimizer][IBM CPLEX]] in
|
||||||
|
particular all offer free academic licenses and work very well with
|
||||||
|
JuMP.
|
||||||
|
|
||||||
|
Another awesome resource is the [[https://neos-server.org/neos/][NEOS Server]]. It offers free computing
|
||||||
|
resources for numerical optimization, including all major free and
|
||||||
|
commercial solvers! You can submit jobs on it in a standard format, or
|
||||||
|
interface your favourite programming language with it. The fact that
|
||||||
|
such an amazing resource exists for free, for everyone is
|
||||||
|
extraordinary. They also have an accompanying book, the [[https://neos-guide.org/][NEOS Guide]],
|
||||||
|
containing many case studies and description of problem types. The
|
||||||
|
[[https://neos-guide.org/content/optimization-taxonomy][taxonomy]] may be particularly useful.
|
||||||
|
|
||||||
|
* Conclusion
|
||||||
|
|
||||||
|
Operations research is a fascinating topic, and it has an abundant
|
||||||
|
literature that makes it very easy to dive into the subject. If you
|
||||||
|
are interested in algorithms, modelling for practical applications, or
|
||||||
|
just wish to understand more, I hope to have given you the first steps
|
||||||
|
to follow, start reading and experimenting.
|
||||||
|
|
||||||
|
* References
|
||||||
Loading…
Add table
Add a link
Reference in a new issue