I checked out this branch and tried to make a command to create a custom doc standard.
It's called like

DogeExport name

where name is the name of the new doc standard, and if it's an existing doc standard the new template will be based on it.

I get errors because b:doge_pattern_single_line_comment is not set, otherwise it works (if I set it). It should default to &commentstring withouth %s, so you should check for it.

fun! DogeExport(name)
  let this_ft = &ft

  " maybe there aren't available doc standards for this filetype
  if !exists('b:doge_patterns')
    let b:doge_patterns = {}
  endif

  " if the given name is an existing doc standard, use it as base for our
  " template, otherwise create an empty one
  if has_key(b:doge_patterns, a:name)
    let template = deepcopy(b:doge_patterns[a:name][0])
    let name = a:name . '_custom'
  else
    let template = {'match': '', 'tokens':'', 'template': [], 'insert': 'above', 'parameters': {'match': '', 'tokens':'', 'format': ''}}
    let name = a:name
  endif

  " get the path for the after/ftplugin file and open it if existing
  " otherwise create a new file with an appropriate path
  let path = ''
  for p in ['~/.vim', '~/vimfiles']
    if isdirectory(expand(p))
      let path = expand(p)
    endif
  endfor
  if has('nvim') && empty(path)
    let path = stdpath('config')
  endif
  if !empty(path)
    let path .= '/after/ftplugin/' . this_ft . '.vim'
  endif
  if filereadable(path)
    exe 'split' escape(path, ' ')
  else
    new
    setfiletype vim
    if !empty(path)
      exe 'file' escape(path, ' ')
    endif
  endif

  " generate the template and paste it at the top of the file
  let doc = []
  call add(doc, '" preserve existing doge settings')
  call add(doc, 'let b:doge_patterns = get(b:, "doge_patterns", {})')
  call add(doc, 'let b:doge_supported_doc_standards = get(b:, "doge_supported_doc_standards", [])')
  call add(doc, 'if index(b:doge_supported_doc_standards, "'.name.'") < 0')
  call add(doc, 'call add(b:doge_supported_doc_standards, "'.name.'")')
  call add(doc, 'endif')
  call add(doc, '')
  call add(doc, '" set the new doc standard as default')
  call add(doc, 'let b:doge_doc_standard = "'.name.'"')
  call add(doc, '')
  call add(doc, '" do not overwrite an existing doc standard')
  call add(doc, 'if !has_key(b:doge_patterns, "'.name.'")')
  call add(doc, 'let b:doge_patterns["'.name.'"] = [{')
  call add(doc, '\  "match": '.string(template.match).',')
  call add(doc, '\  "tokens": '.string(template.tokens).',')
  if has_key(template, 'parameters')
    call add(doc, '\  "parameters": {')
    call add(doc, '\    "match": '.string(template.parameters.match).',')
    call add(doc, '\    "tokens": '.string(template.parameters.tokens).',')
    call add(doc, '\    "format": '.string(template.parameters.format).',')
    call add(doc, '\  },')
  endif
  call add(doc, '\  "template": '.string(template.template).',')
  call add(doc, '\  "insert": '.string(template.insert).',')
  call add(doc, '\}]')
  call add(doc, 'endif')
  call add(doc, '')
  call setreg('"', doc)
  1
  normal! P'[=']
endfun

command! -nargs=1 -complete=customlist,doge#command_complete DogeExport call DogeExport(<q-args>)

Anyway there's a problem: every time the file is loaded with e and the ftplugins are loaded, the b:doge_supported_doc_standards variable is flooded with repeated entries. I'm not going to fight with this.

Just try to load a supported filetype and do

:edit

a couple of times and you'll see.

It should default to &commentstring withouth %s, so you should check for it.

We can do that and %s can be replaced with the regex pattern.\{-} just like other existing variables.

Also, what do you mean with so you should check for it? Add a fallback to &commentstring when b:doge_patterns_single_line_comment is not set?

I suggest you change

if !exists('b:doge_patterns')
  let b:doge_patterns = {}
endif

to

let b:doge_patterns = doge#buffer#get_patterns()

since this will take tests into account and already adds proper fallbacks if it doesn't exists.

[...] the b:doge_supported_doc_standards variable is flooded with repeated entries.

Fixed it, see 7db29f7

Add a fallback to &commentstring when b:doge_patterns_single_line_comment is not set?

Yes, instead of throwing unhandled errors.

I suggest you change

The code that command generates goes into the user's ftplugin, it doesn't need tests nor fallbacks. A custom doc standard will be created, that isn't handled by doge in any way, except that it's added to the supported doc standards and it can be used by DogeGenerate, but doge should have no other business with it.

It will neither overwrite a default doc standard, it will generate an entry like numpy_custom instead.

The code that command generates goes into the user's ftplugin, it doesn't need tests nor fallbacks. A custom doc standard will be created, that isn't handled by doge in any way, except that it's added to the supported doc standards and it can be used by DogeGenerate, but doge should have no other business with it.

Clear. So your DogeExport command will basically resolve #60 with more ease for the user, right?

I will add a test later for DogeExport when you're done.

Also, why name it DogeExport? Why not something like DogeCreateDocStandard ? It's a bit longer, but described what it does. You're not exporting anything from my point of view.

Why not something like DogeCreateDocStandard ?

Yes it's better, it was just a temporary name.

So your DogeExport command will basically resolve #60 with more ease for the user, right?

It's easier than looking for a source for the doc standard, but it will create an empty template unless you use an existing one as base (the command argument). It should be documented how to properly make a custom doc standard, what the dictionary entries do, etc.

It's easier than looking for a source for the doc standard, but it will create an empty template unless you use an existing one as base (the command argument). It should be documented how to properly make a custom doc standard, what the dictionary entries do, etc.

Agree, but documentation will come after it's done and works as expected. I also took note in the TODO list of this PR of the following: Validate pattern for required keys. I will do that right away now, since this is also something that should be done to prevent the user from creating invalid keys.

You shouldn't be overzelous on that, the template generates all keys, if the user wants to make mistakes, let him learn from his mistakes. Instead the code should check for things that template doesn't generate, things like b:doge_pattern_single_line_comment.

[...] if the user wants to make mistakes, let him learn from his mistakes.

Getting a big error and figuring it out as a regular user takes more time instead of seeing 1 clear message with what to do and exit the generation early. UX-wise, it's better to catch the error.

EDIT: Let's skip this part for now.

@mg979 I renamed generate.vim to pattern.vim since this makes more sense. I used generate.vim at the beginning, assuming I'd create different types of generations, but this is not the case anymore.

@mg979 Also, why should I take care for a fallback for b:doge_patterns_single_line_comment? You can also add this when running DogeCreateDocStandard, right?

why should I take care for a fallback for b:doge_patterns_single_line_comment? You can also add this when running DogeCreateDocStandard, right?

Why would you want the user to keep that sort of stuff in their own ftplugin, only the strictly necessary stuff should go there, that variable is not.

Anyway you just need to use get() and the fallback value when you use the variable.

Why would you want the user to keep that sort of stuff in their own ftplugin, only the strictly necessary stuff should go there, that variable is not.

They do not need to define it since it's already defined. If doge defines it already in ftplugin/python.vim then if you echo it in ~/.vim/after/ftplugin/python.vim it will echo it. The reason why it errors for you is that you have a typo in b:doge_patterns_single_line_comment, it should be doge_pattern and not doge_patterns (note the s at the end). You should use: b:doge_pattern_single_line_comment.

Sorry for the typo, but the reason was that I was trying the command on vimL, that doesn't have that variable because it's not supported. That check is necessary for the languages that aren't supported (another good reason why it should be handled by doge, because it would be a repetition, and possibly conflicting, among ftplugins). Anyway, what can be handled by doge should be handled by doge.. users don't want obscure stuff in their ftplugins, me included.

[..] That check is necessary for the languages that aren't supported (another good reason why it should be handled by doge, because it would be a repetition, and possibly conflicting, among ftplugins). Anyway, what can be handled by doge should be handled by doge.. users don't want obscure stuff in their ftplugins, me included.

It would be a possible fallback, but I highly doubt whether to implement this. I don't see any reason to make this an optional variable. I want to require the variable, because if you take PHP, there are more versions of creating single line comments: /* ... */ and // ... and &commentstring does not cover everything and you also are required to specify b:doge_pattern_multi_line_comment as well.

There are a minimum amount of variables to create a proper doc standard and the ones I just described about the comment are one of the required ones. If Vim has more variables I am not aware of that would cover these variables, I'd more than happy to implement those to save code.

As you wish. But I think you're hurting your own plugin, thinking like this. Those variables contain patterns that aren't so obvious and could be handled by doge, in a couple lines of code. And only for unsupported languages, because for languages that have already a doc standard in doge, this won't even matter.

Edit: the command could be changed to add those variables only in the case that the filetype is unsupported, will do it maybe later.

As you wish. But I think you're hurting your own plugin, thinking like this. Those variables contain patterns that aren't so obvious and could be handled by doge, in a couple lines of code. And only for unsupprted languages, because for languages that have already a doc standard in doge, this won't even matter.

Fair enough. If it's for unsupported doc standards then it's up to the user in my opinion.

And what about b:doge_pattern_multi_line_comment? Do you want this variable also to fallback on &commentstring ? I don't think we have another choice.

The PHP scenario is a specific scenario. There is a class definition that does not have nor need a parameters key. The propertyName in this case is used in the preprocess function

Understood. So you want custom patterns to be able to use preprocessors as well? And the command should assume that when a token is found that has not a key, it's handled by a preprocessor? It gets complicated if this basically requires browsing doge codebase to see what these prerocessors do.
I think support for funcrefs in templates would be better, for custom patterns.

What do you exactly mean with funcrefs ?

For example I have this in after/ftplugin/vim.vim

" preserve existing doge settings
let b:doge_patterns = get(b:, "doge_patterns", {})
let b:doge_supported_doc_standards = get(b:, "doge_supported_doc_standards", [])
if index(b:doge_supported_doc_standards, "vimdoc1") < 0
  call add(b:doge_supported_doc_standards, "vimdoc1")
endif
if index(b:doge_supported_doc_standards, "vimdoc2") < 0
  call add(b:doge_supported_doc_standards, "vimdoc2")
endif

" doge uses these patterns to identify comments, change if needed
let b:doge_pattern_single_line_comment = '\V"\.\{\-\}\$'
let b:doge_pattern_multi_line_comment = '\V"\.\{\-\}\$'

" set the new doc standard as default
let b:doge_doc_standard = "vimdoc1"

let s:main_pattern       = '\m^fu[ction]*!\?\s\+\([[:alnum:]_:.#]\+\)\s*(\(.\{-}\))'
let s:funcname_pattern   = '\m^fu[ction]*!\?\s\+\([[:alnum:]_:.#]\+\)'
let s:parameters_pattern = '\m\([^,]\+\)'

fun! s:return_value()
  return search('\C\<return\> \p\+', 'n', search('^endfu', 'n')) ?
        \ '" Returns: !return' : '-'
endfun

" do not overwrite an existing doc standard
if !has_key(b:doge_patterns, "vimdoc1")
  let b:doge_patterns["vimdoc1"] = [{
        \ "match": s:main_pattern,
        \ "tokens": ['funcname', 'parameters'],
        \ 'funcname': {
        \   'match': s:funcname_pattern,
        \   'tokens': ['name'],
        \   "format": '{name} !description',
        \ },
        \ "parameters": {
        \   'match': s:parameters_pattern,
        \   'tokens': ['name'],
        \    "format": "@param {name}: !description",
        \ },
        \ 'template': [
        \   '""'.repeat('=', 77),
        \   '%(funcname|" Function: {funcname})%',
        \   '" !summary',
        \   "%(parameters|\" {parameters})%",
        \   function('s:return_value'),
        \   '""'.repeat('=', 77),
        \   '""'],
        \ "insert": 'above',
        \}]
  let b:doge_patterns["vimdoc2"] = [{
        \ "match": s:main_pattern,
        \ "tokens": ['funcname', 'parameters'],
        \ 'funcname': {
        \   'match': s:funcname_pattern,
        \   'tokens': ['name'],
        \   "format": '{name} !description',
        \ },
        \ "parameters": {
        \   'match': s:parameters_pattern,
        \   'tokens': ['name'],
        \    "format": "@param {name}: !description",
        \ },
        \ 'template': [
        \   '"',
        \   '" !summary {{{1',
        \   '" !description',
        \   "%(parameters|\" {parameters})%",
        \   function('s:return_value')],
        \ "insert": 'below',
        \}]
endif

You see, in the template key, I'm using a funcref that is evaluated when DogeGenerate is run, so that it adds the Returns: line only if the function returns something specific.

Funcrefs woul need to be supported in doge#pattern#generate, though:

-  for l:line in a:pattern['template']
+  for l:Line in a:pattern['template']
+    " check the type of the current line, it could be a funcref
+    let l:line = type(l:Line) == 2 ? l:Line() : l:Line

I don't know if your preprocessors serve the main purpose, but for custom doc standards, a funcref is much better, you keep it in your file and you know what it is doing.

It looks way too complicated. I rather add a preprocess function with a few lines instead of creating all kinds of functions in my file. Every variable in the ftplugin that is added by a preprocess function is commented, so there's no problem here.

How's your CreateDogeDocStandard going? Are you done already? Aiming for this PR to be merged soon.

Also the regex fu[ction]* is not matching what you want and the amount of possibles this regex can match is infinity. See fiddle. You should do: fun\(ction\)\? or \(fun\|function\)

Updated the command.

It looks way too complicated. I rather add a preprocess function with a few lines instead of creating all kinds of functions in my file

Funcrefs aren't for your files, but for custom doc standards: there the problem is reversed, for someone who knows nothing about doge preprocessors, and just wants a custom standard in his own ftplugin file.

Edit: command fixed again

the regex fu[ction]* is not matching what you want and the amount of possibles this regex can match is infinity. See fiddle. You should do: fun\(ction\)\? or \(fun\|function\)

I know but this recognizes fu, fun, func etc I was just lazy to write all combinations up to function, it can be fixed, but since doge checks the current line and you trigger it yourself, there's not much room for error.

Use \m instead of \V to prevent portability issues.

I'll think about your suggestion.

Use \m instead of \V to prevent portability issues.

Updated the command.

Can you fork the repo, create a sub-branch from feature/rewrite-ftplugin-structure with a working version and then make a PR to this branch as well? This helps with testing things and then we can discuss your code in your PR.

Also, why is this 1 there?

  " ...
  if has_key(template, 'parameters')
    call add(doc, '\  "parameters": {')
    call add(doc, '\    "match": '.string(template.parameters.match).',')
    call add(doc, '\    "tokens": '.string(template.parameters.tokens).',')
    call add(doc, '\    "format": '.string(template.parameters.format).',')
    call add(doc, '\  },')
  endif
  call add(doc, '\  "template": '.string(template.template).',')
  call add(doc, '\  "insert": '.string(template.insert).',')
  call add(doc, '\}]')
  call add(doc, 'endif')
  call add(doc, '')
  call setreg('"', doc)
  1 " <-- this .. should be here?
  normal! P'[=']
endfun

command! -nargs=1 -complete=customlist,doge#command_complete DogeExport call DogeExport(<q-args>)

1 goes to the top of the file, like normal! gg. The function should go to its own autoloaded file or somewhere in particular?

The function should go to its own autoloaded file or somewhere in particular?

Yes. Maybe in pattern.vim where the name would be doge#pattern#create().

I made a PR for this command, I tested it in vimL and php and it seems to work ok.

Another thing I would like to change (but not sure if this is the way to go) is instead of making let b:doge_patterns. a list, I think we should make these dictionaries as well and use named keys.

Thinking about it again, to preserve the ability to try patterns in a defined order, one could make them like:

let b:doge_pattern[doc_standard] = {
      \ 'types': ['function'],
      \ 'patterns': {
      \   'function': { ... }
      \  }
      \}

and then process them in the order defined by b:doge_patterns[doc_standard].types.
Though I know it's a lot of work by now.

Thinking about it again, to preserve the ability to try patterns in a defined order, one could make them like:

let b:doge_pattern[doc_standard] = {
      \ 'types': ['function'],
      \ 'patterns': {
      \   'function': { ... }
      \  }
      \}

and then process them in the order defined by b:doge_patterns[doc_standard].types.
Though I know it's a lot of work by now.

I wouldn't do that and I don't think this is a need. The array already defines the order. If you want to change them, re-order the array.

@mg979 Thanks for your huge contribution. It's been appreciated!

This feature has been merged and released in v2.0.0.

Feel free to submit any new issues if you experience any unwanted behavior in the future. Thanks for your contribution!