Elamur Research⌕
☰
home / papers / 2021 / Using a New Nonlinear Gradient Method for Solving Large Scale Convex Optimization Problems with an Application on Arabic Medical Text

Using a New Nonlinear Gradient Method for Solving Large Scale Convex Optimization Problems with an Application on Arabic Medical Text

Summary

The authors propose a new nonlinear conjugate gradient (CG) method for large unconstrained convex minimization. Its CG coefficient, called AWHM, is a weighted mix of two existing coefficients, HRM and NHS. The mixing weight θ is recomputed at every iteration so that the new direction satisfies a Dai–Liao-style conjugacy condition. If θ falls at either end of [0,1], the method reverts to one of the parent formulas. The authors prove that the directions give sufficient descent and that the method converges globally under standard assumptions. On a benchmark of 110 test functions, they compare it with HRM and NHS by iteration count and runtime, using Dolan–Moré performance profiles. They then use it as the optimizer for the classification head when fine-tuning BERT for named entity recognition on an Arabic medical corpus. Compared with the earlier Adam-based setup, F1 stayed stable and training time dropped. The work matters mainly as a demonstration that a classical CG optimizer can be plugged into a modern NLP fine-tuning pipeline.

Key points

  • AWHM is defined as (1−θ_k)·β^NHS + θ_k·β^HRM, with θ_k taken from a conjugacy equation. The step size comes from a strong Wolfe-Powell line search (δ = 10⁻⁴, σ = 0.9).
  • Two safeguards are built in:
    • a restart to steepest descent whenever |g_{k+1}ᵀg_k| ≥ 0.2‖g_{k+1}‖²;
    • rescaling of the step size by the ratio of successive direction norms.
  • The method solved 109 of 110 standard problems drawn from the Andrei and Mishra collections. It is compared with HRM and NHS by iterations and execution time.
  • The NER data come from three volumes of the Arab Medical Encyclopedia (Syria):
    • Respiratory: 28 articles, 6691 sentences.
    • Cardiovascular: 33 articles, 9464 sentences.
    • Skin: 22 articles, 5921 sentences.
    • Medical students annotated four entity types (disease, organ, symptom, drug) in BIO format.
  • Baseline BERT+Adam F1 scores were 87.71 (disease), 86.30 (organ), 69.85 (symptoms) and 77.50 (drug). The new-optimizer F1 values are missing from the extracted text; the authors describe them as stable.
  • Training time fell from 9.25 h to 7.5 h on the same machine (batch 32, 4 epochs), which the authors report as an 18.9% improvement.

Interesting ideas

  • I1. Adaptive blending of two update rules. Two update rules are combined with a per-step weight solved from a desired property, and the method falls back to a pure parent rule when the weight leaves [0,1]; this is a principled hybrid rather than a fixed average. — For Mova (learn): when merging votes from several witness parsers into silver data, the mixing weight could be derived per sentence from an agreement constraint. Clipping would then fall back to a single trusted witness.
  • I2. Restart test on stale momentum. The method discards accumulated momentum and takes a plain gradient step whenever consecutive gradients become too aligned; this is a cheap guard against accumulated history pulling the search off course. — For Mova (learn): perceptron or gparse training could reset averaged or momentum state when successive update vectors become highly correlated. This would help avoid oscillation on noisy silver batches.
  • I3. Swapping only the head's optimizer. The authors change the optimizer of the small classification head while leaving the pretrained encoder setup untouched, which isolates the effect of the change. — For Mova (tree): gparse experiments could vary only the arc/label scorer's training recipe over a fixed encoder. Any gain or loss would then be clearly attributable to that change.
  • I4. Performance profiles across many test problems. Dolan–Moré profiles summarize how often, and by what factor, each method stays close to the best result over a large problem set, instead of reporting one average. — For Mova (learn): comparing parser or repair-rule variants across many treebanks and test suites with such profiles would show robustness, not just mean LAS.
  • I5. Controlled reporting of cost and quality. Wall-clock cost is reported next to per-category F1 under identical hardware, batch size and epochs, so speed and quality trade-offs stay visible per category. — For Mova (answer): evaluations of readers on school tests or word problems could report accuracy per question type together with runtime under fixed settings. This would reveal where a faster variant loses quality.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Named entity recognition

Cited by

1 paper of our library cites it (of the 140884 whose bibliographies we read); each has a page here.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2106.04383 · arXiv 2106.04383

@misc{hammoud2021using,
  title = {{Using a New Nonlinear Gradient Method for Solving Large Scale Convex Optimization Problems with an Application on Arabic Medical Text}},
  author = {Hammoud, Jaafar and Eisa, Ali and Dobrenko, Natalia and Gusarova, Natalia},
  year = {2021},
  doi = {10.48550/arxiv.2106.04383},
  eprint = {2106.04383},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2106.04383}
}

References

28 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. HAN, X., ZHANG, J., CHEN,J., (2017) “A New Hybrid Conjugate Gradient Algorithm for Unconstrained Optimization” Bulletin of the Iranian Mathematical Society Vol. 43, No. 6, pp. 2067-2084. find
  2. HAMODA, M ., MAMAT, M., RIVAIE, M., SALLEH, Z., (2016) “A Conjugate Gradient Method with Strong Wolfe-Powell Line Search for Unconstrained Optimization” Applied Mathematical Sciences, Vol. 10, No. 15, pp.721 – 734. find
  3. SALIH, Y., HAMODA, M., RIVAIE, M., (2018) “New Hybrid Conjugate Gradient Method with Global Convergence Properties for Unconstrained Optimization” Malaysian Journal of Computing and Applied Mathematics, Vol .1,No1- pp.29-38. find
  4. P. Wolfe, (1971) “Convergence conditions for ascent methods. II: some corrections”, SIAM Review, vol. 13, no. 2, pp. 185-188. find
  5. HESTENES M.R.; STIEFEL E.,(1952) “Method of Conjugate Gradient for Solving Linear Equations” Journal of Research of the National Bureau of Standards, Vol. 49, No. 6, pp. 409–436. find
  6. FLETCHER R. ; REEVES C., (1964) “Function Minimization by Conjugate Gradients” the Computer Journal, Vol. 7, No. 2, pp.149-154. find
  7. POLAK E.; RIBIÈRE G., (1969) “Note Sur la Convergence de Méthodes de Directions Conjuguées” Revue Françoise d’Informatique et de Recherché Opérationnelle, Vol.16, No.3, pp. 35–43. find
  8. B. T. Polyak, (1969) “The conjugate gradient method in extreme problems” USSR Computational Mathematics and Mathematical Physics, 9 , 94-112. find
  9. GHAZALI, K., SULAIMAN, J., DASRIL, Y., GABDA, D., (2019) “Newton-MSOR Method for Solving Large-Scale Unconstrained Optimization Problems with an Arrowhead Hessian Matrices” Transactions on Science and Technology, Vol. 6, No. 2-2,pp. 228 – 234. find
  10. BERAHAS, A.S., TAK_A_C, M., (2020) “A robust multi-batch l-bfgs method for machine learning.” Optimization Methods and Software, Vol. 35, No.1, pp. 191-219. find
  11. BERAHAS, A.S., JAHANI, M., TAK_AC, M., (2019) “Quasi-Newton Methods for deep learning: Forget the past, just sample.” arXiv preprint arXiv: 1901.09997 ↗
  12. Khanaiah, Z. and Hmod, G., (2017) “Novel hybrid algorithm in solving unconstrained optimizations problems.” Int. J. Novel Res. Phys. Chem. Math., 4(3), pp.36-42 find
  13. ERWAY, J.B., GRI_N, J., MARCIA, R.F., OMHENI, R., (2019) “Trust-Region Algorithms for Training Responses: Machine Learning Methods Using Indefinite Hessian Approximations.” Optimization Methods and Software pp. 1-28. find
  14. KIMIAEI M., GHADERI S., (2017) “A New Restarting Adaptive Trust–Region Method for Unconstrained Optimization.” Journal of the Operations Research Society of China, 5(4),487–507. find
  15. DAWAHDEH, M., et al., (2020) “A New Spectral Conjugate Gradient Method with Strong Wolfe-Powell Line Search.” International Journal of Emerging Trends in Engineering Research,Vol.8,No.2, pp.391 – 397. find
  16. ABDULLAHI,I., AHMAD, R., (2017) “Global Convergence Analysis of A New Hybrid Conjugate Gradient Method for Unconstrained Optimization Problems.” Malaysian Journal of Fundamental and Applied Sciences Vol. 13, No. 2,pp. 40-48. find
  17. Dai, Y., H., Liao L Z., (2001) “New conjugacy conditions and related nonlinear conjugate gradient methods.” Appl Math Optim, , Vol.43, pp.87–101. find
  18. RIVAIE, M., MAMAT, M., ABASHAR, A., (2015) “A new class of nonlinear conjugate gradient coefficients with exact and inexact line searches.” Appl. Math. Comp.268, pp. 1152-1163. find
  19. N. Andrei, (2008) “An unconstrained optimization test functions collection.” Advanced Modelling and Optimization, 10, 147-161. find
  20. S. Mishra, (2007) “Some new test functions for global optimization and performance of repulsive particle swarm method.” Munich Personal RePEc Archive, Apr 13. find
  21. Dolan, E.D., More, J.J., (2002) “Benchmarking optimization software with performance profiles.” Mathematical Programming, 91 (2), 201-213. find
  22. Kingma, D.P. and Ba, J., (2014) “Adam: A method for stochastic optimization.” arXiv preprint arXiv:1412.6980. ↗
  23. Sutskever, I., Martens, J., Dahl, G. and Hinton, G., (2013) “On the importance of initialization and momentum in deep learning.” In International conference on machine learning (pp. 1139-1147). PMLR. find
  24. Tieleman, T. and Hinton, G., (2012) “Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude.” COURSERA: Neural networks for machine learning, 4(2), pp.26-31. find
  25. Hammoud, J., et al., (2020) “Named entity recognition and information extraction for Arabic medical text”, Multi Conference on Computer Science and Information Systems, MCCSIS 2020 - Proceedings of the International Conference on e-Health 2020, 121-127. find
  26. Devlin, J., Chang, M.W., Lee, K. and Toutanova, K., (2018) “Bert: Pre-training of deep bidirectional transformers for language understanding.” arXiv preprint arXiv:1810.04805. ↗
  27. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). “Attention is all you need.” arXiv preprint arXiv:1706.03762. ↗
  28. Al-Najjar, H., Dabash, M., Al-Issa, A., (2020) "A Nonlinear Gradient Method for Solving Convex Functions with Large Scale", Al-Baath University Journal for Research and Scientific Studies, Volume (42). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hammoud-2021-using-new-nonlinear-gradient-method/