Hello guys,

I have a question regarding the nested backquotes in macros. I wrote a macro, which creates lexical bindings for “port:ip” values:

(defun mkstr (&rest args)
  (with-output-to-string (s)
    (dolist (a args) (princ a s))))

(defun mksymb (&rest args)
  (values (intern (string-upcase (apply #'mkstr args)))))

;; my macro
(defmacro with-free-ports (start end &body body)
  (let ((range (loop for port from start to end collect (format NIL "127.0.0.1:~a" port)))
	(n 0))
    `(let ,(mapcar #'(lambda (p) `(,(mksymb "PORT-" (incf n)) ,p)) range)
       (progn ,@body))))

One sets a range of ports on localhost and these ports are bound to symbols port-1, port-2, etc…

(with-free-ports 1 3 port-1) ;; => "127.0.0.1:1"

This works fine if the start or end parameters are given as values. But if they are variables. which must be evaluated, this macro doesn’t work:

(let ((start 1))
  (with-free-ports start 3 port-1)) ;; error

In order to fix it, I made the let- bindings a part of the macro-expansion:

(defmacro with-free-ports (start end &body body)
  `(let ((range (loop for port from ,start to ,end collect (format NIL "127.0.0.1:~a" port)))
	(n 0))
     `(let ,(mapcar #'(lambda (p) `(,(mksymb "PORT-" (incf n)) ,p)) range)
	       (progn ,@body))))

but get a compilation warning that the body is never used. I assume this is because of the inner backquote.

To evaluate ,@body inside the inner backquote, I use one more comma, and the macro compiles without warnings:

(defmacro with-free-ports (start end &body body)
  `(let ((range (loop for port from ,start to ,end collect (format NIL "127.0.0.1:~a" port)))
	(n 0))
     `(let ,(mapcar #'(lambda (p) `(,(mksymb "PORT-" (incf n)) ,p)) range)
	       (progn ,,@body)))) ;; one more comma here

But it doesn’t work:

(let ((start 1))
  (with-free-ports start 3 port-1)) ;; error: port-1 is unbound

because with this ,,@body I evaluate port-1: (progn ,port-1) and this triggers the error.

I would appreciate if smbd can help me a bit and say what I am doing wrong.

Thank you.

  • dr675r@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    11 months ago

    I will preface this by saying in general I don’t think the approach of interning symbols and binding them at runtime is a good idea. If you’ve referenced these symbols by name in the body of the form, you already knew you needed them, so why not just just use LET to create the binding? Without knowing the details of your use case it sounds like it could be an XY problem.

    However, this being Common Lisp, there is a way to do most things and the special operator PROGV is used to create dynamic variable bindings. This is not a tool I reach for often, but it does occasionally come in handy. I am not proud of this:

    (in-package #:cl-user)
    (ql:quickload "alexandria")
    
    (defmacro with-free-ports ((start end &key (prefix "PORT-") limit) &body body)
      (check-type limit (or null (integer 1 *)))
      (alexandria:with-gensyms (first-port last-port index vars values symbol-prefix)
        `(let ((,first-port ,start)
               (,last-port ,end)
               (,symbol-prefix ,prefix))
           (check-type ,first-port (integer 1 *))
           (check-type ,last-port (integer 1 *))
           (assert (>= ,last-port ,first-port) (,first-port ,last-port) "port range not ordered")
           (check-type ,symbol-prefix (or string symbol character))
           ,(when limit
              `(when (>= (- ,last-port ,first-port) ,limit)
                 (error ,(format nil "WITH-FREE-PORTS limit (~A) exceeded" limit))))
           (loop named with-free-ports
                 for ,index from ,first-port to ,last-port               
                 collecting (alexandria:symbolicate ,symbol-prefix (princ-to-string ,index)) into ,vars
                 collecting (format nil "127.0.0.1:~A" ,index) into ,values
                 finally (return-from with-free-ports (progv ,vars ,values ,@body))))))
    

    To use it:

    (let ((start 80))
      (with-free-ports (start (+ start 5) :limit 10)
        (setf port-80 "http")
        (values port-80 port-85))) ;; => "http", "127.0.0.1:85"
    

    I have added what I consider the bare minimum of safety checks, such as a limit on the number of symbols it will create. You also lose help from the compiler which can’t warn you about unused variables and you may get warned that some of your variables have been assumed special (at least on LispWorks). Overall, I think its a bit of hack and not something I would put in production code.

    It is also worth noting the following loop achieves the same thing, without the hackery and is trivial to change to a vector if you’re worried about efficiency:

    (let ((start 80))
      (loop for port from start to (+ start 5)
            collecting (format nil "127.0.0.1:~A" port) into ports
            finally (progn
                      (setf (first ports) "http")
                      (return (values (first ports) (sixth ports))))))
    
    • xhash101@alien.topOPB
      link
      fedilink
      English
      arrow-up
      1
      ·
      11 months ago

      Thanks a lot!

      If you’ve referenced these symbols by name in the body of the form, you already knew you needed them,

      Yes, but I do not know how many of them. The purpose of this macro is to create a lexical environment to test a network code. Some functions take just one port as a parameter, but the others - two and more. With this macro it would be very easy to call these functions and to link their IO through the port numbers:

      (with-free-ports 0 10
        (fn-1 port-1 port-2 port-3)
        (fn-2 port-1))
      

      And with the first posted version of with-free-ports I achieved it. The problems appeared when I tried to bind start in run-time. From your reply and from the u/lispm comments I understood that this is not a good idea. What I do not understand, is how to use such macros inside another macros/functions, which can supply with-free-ports with start and end parameters ? Also, I cannot figure out, when exactly macro-expansions take place if I use many enclosed macros. Are they expanded all at once when I compile my code?

      Anyway, I am very grateful for your suggestions. This helps a lot to learn CL.

      • dr675r@alien.topB
        link
        fedilink
        English
        arrow-up
        1
        ·
        11 months ago

        Macros are expanded recursively, so if the result of expanding a macro is itself a macro then the evaluator or compiler will immediately expand it again. This is why we have MACROEXPAND-1 which expands it once, and MACROEXPAND which expands a form until it is no longer a macro form. I would suggest you MACROEXPAND-1 my answer, then repeat with MACROEXPAND to see the non-macro code that actually ends up being compiled/evaluated.

        You may find Section 3 of the HyperSpec helpful. The spec goes into a fair amount of detail about the semantics of both evaluation and compilation which should help.

  • tdrhq@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    11 months ago

    Try not to fight so much with macros. I see what you’re trying to do, but it might be better to just bind a list or array of ports, instead of binding port{1-n}.

    If you agree to use an array for binding, then you can use https://github.com/tdrhq/easy-macros to write this macro without using any backquotes at all. The macro definition would look something like:

    (def-easy-macro with-free-ports (start end &binding ports &fn fn)
      (unwind-protect
         (fn (... construct list or array of ports ...))
        (cleanup #| this cleanup is probably the reason you're using macros in the first place |#)))
    

    And you’ll call it as:

    (with-free-ports (1000 2000 port)
      (... do something with port ...))
    
  • lispm@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    11 months ago

    If you compile code, then the compiler expands macro forms at compile-time. If you want to generate a variable number of let bindings, then the number needs to be known at compile-time.

    start and end thus can’t be variables at run-time.

    (let ((start 1))
      (with-free-ports start 3 port-1))
    

    If we compile above form, then at compile-time the value of start generally is unknown.

    • xhash101@alien.topOPB
      link
      fedilink
      English
      arrow-up
      1
      ·
      11 months ago

      Thank you for your reply.

      The compiler generally will not execute the LET and then compile the WITH-FREE-PORTS form with the new binding

      Here is an example when the compiler does exactly that:

      (defmacro with-free-ports (start end &body body)
        `(list ,start ,end ,@body))
      
      (let ((start 1)
            (end 3))      
        (with-free-ports start end NIL))
      

      So, why does it work?

      • lispm@alien.topB
        link
        fedilink
        English
        arrow-up
        1
        ·
        11 months ago

        No, the compiler does not do that. If you look into your macro here, it just puts the args start and end back in. The generated code then is executed. Remember: START and END are bound to the source forms, not computed values.

        In the original macro you had the form (loop for port from start to end ...), which you tried to compute at macro-expansion time. But the values of START and END are not necessary numbers, but source forms, like variable names.